librelist archives

« back to archive

syncing an IMAP mailbox

syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 07:08
I am trying to figure out the procedure for syncing an IMAP folder. This 
is what I have come up with:

1. Call messagesFromUID and retrieve all data in all messages 
(CTFetchAttrEnvelope | CTFetchAttrBodyStructure). Save it all to disk and 
record the max UID.
2. When you want to update
	a. call messagesFromUID and retrieve all data from the current maxUID + 1
to 0 (the last message). This will get all messages that have come in 
since your last update. Store them as in step #1.
	b. call messagesFromUID with CTFetchAttrDefaultsOnly and retrieve the 
flags for all messages. This is the only data that could have changed 
since the message was first downloaded. Compare the flags to the existing 
data and update anything that has changed.

Does that sound correct? Is there a more efficient way to do it?

Re: [mailcore] syncing an IMAP mailbox

From:
Jan Chaloupecky
Date:
2012-07-30 @ 07:16
Just some quick thoughts:


>         a. call messagesFromUID and retrieve all data from the current
> maxUID + 1 to 0 (the last message). This will get all messages that have
> come in since your last update. Store them as in step #1.



The UID is not sequential so the maxUID + 1 won't work I guess.



Your algorithm will only fetch new e-mail (which are not yet stored
locally) but it will not handle emails that were deleted remotely.





On Mon, Jul 30, 2012 at 9:08 AM, Rick Gigger <rgigger@gmail.com> wrote:

> I am trying to figure out the procedure for syncing an IMAP folder. This
> is what I have come up with:
>
> 1. Call messagesFromUID and retrieve all data in all messages
> (CTFetchAttrEnvelope | CTFetchAttrBodyStructure). Save it all to disk and
> record the max UID.
> 2. When you want to update
>         a. call messagesFromUID and retrieve all data from the current
> maxUID + 1 to 0 (the last message). This will get all messages that have
> come in since your last update. Store them as in step #1.
>         b. call messagesFromUID with CTFetchAttrDefaultsOnly and retrieve
> the flags for all messages. This is the only data that could have changed
> since the message was first downloaded. Compare the flags to the existing
> data and update anything that has changed.
>
> Does that sound correct? Is there a more efficient way to do it?
>



-- 
Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 07:30
On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> Just some quick thoughts:
>  
>         a. call messagesFromUID and retrieve all data from the current 
maxUID + 1 to 0 (the last message). This will get all messages that have 
come in since your last update. Store them as in step #1.
> 
> 
> The UID is not sequential so the maxUID + 1 won't work I guess.

But if you give it number that isn't there won't it just give you the next
highest one that does exist as the first item? If not I could just give it
maxUID then and always throw that one out when it comes, but that seems 
inefficient. What is the best way to say, give me all the messages from 
this mailbox that I don't have yet?

> 
> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.

In 2.b. when I request everything in the mailbox I could assume that any 
message that I have already downloaded but is not in the results has been 
deleted from the mailbox.

All of this seems very inefficient though. Does IMAP specify a specific 
way to handle syncing? I've noticed that Mail.app will reflect changes 
that I make in the gmail web interface within about 5 seconds. It can't 
possibly be checking and rechecking over 32k messages every few seconds? 
Do you know how mail clients do this? It seems like there would be a 
standard way of doing this. If not maybe I could set up my own IMAP server
and some logging and see what it's doing.

Re: [mailcore] syncing an IMAP mailbox

From:
Jan Chaloupecky
Date:
2012-07-30 @ 07:41
I'm also not sure what is the best way to handle this. Maybe I will analyze
the Mail.app traffic to see what it is doing.

To get the new mails, I would fetch messages from local maxUID (included)

To sync, I would define a user configurable option, eg "Days to Sync", look
in local database to find the last mail matching that option, then fetch
messages from lastMailToSyncUID.


But you are right, this does not seem to be efficient if I want to keep all
mail synced. It looks like we'll have to read the IMAP good practices :)





On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:

> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>
> Just some quick thoughts:
>
>
>>         a. call messagesFromUID and retrieve all data from the current
>> maxUID + 1 to 0 (the last message). This will get all messages that have
>> come in since your last update. Store them as in step #1.
>
>
>
> The UID is not sequential so the maxUID + 1 won't work I guess.
>
>
> But if you give it number that isn't there won't it just give you the next
> highest one that does exist as the first item? If not I could just give it
> maxUID then and always throw that one out when it comes, but that seems
> inefficient. What is the best way to say, give me all the messages from
> this mailbox that I don't have yet?
>
>
> Your algorithm will only fetch new e-mail (which are not yet stored
> locally) but it will not handle emails that were deleted remotely.
>
>
> In 2.b. when I request everything in the mailbox I could assume that any
> message that I have already downloaded but is not in the results has been
> deleted from the mailbox.
>
> All of this seems very inefficient though. Does IMAP specify a specific
> way to handle syncing? I've noticed that Mail.app will reflect changes that
> I make in the gmail web interface within about 5 seconds. It can't possibly
> be checking and rechecking over 32k messages every few seconds? Do you know
> how mail clients do this? It seems like there would be a standard way of
> doing this. If not maybe I could set up my own IMAP server and some logging
> and see what it's doing.
>



-- 
Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 07:54
I'm checking this out to see if it has any relevant info:

http://www.imapwiki.org/ClientImplementation


On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> I'm also not sure what is the best way to handle this. Maybe I will 
analyze the Mail.app traffic to see what it is doing.
> 
> To get the new mails, I would fetch messages from local maxUID (included)
> 
> To sync, I would define a user configurable option, eg "Days to Sync", 
look in local database to find the last mail matching that option, then 
fetch messages from lastMailToSyncUID.
> 
> 
> But you are right, this does not seem to be efficient if I want to keep 
all mail synced. It looks like we'll have to read the IMAP good practices 
:)
> 
> 
> 
> 
> 
> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> Just some quick thoughts:
>>  
>>         a. call messagesFromUID and retrieve all data from the current 
maxUID + 1 to 0 (the last message). This will get all messages that have 
come in since your last update. Store them as in step #1.
>> 
>> 
>> The UID is not sequential so the maxUID + 1 won't work I guess.
> 
> But if you give it number that isn't there won't it just give you the 
next highest one that does exist as the first item? If not I could just 
give it maxUID then and always throw that one out when it comes, but that 
seems inefficient. What is the best way to say, give me all the messages 
from this mailbox that I don't have yet?
> 
>> 
>> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.
> 
> In 2.b. when I request everything in the mailbox I could assume that any
message that I have already downloaded but is not in the results has been 
deleted from the mailbox.
> 
> All of this seems very inefficient though. Does IMAP specify a specific 
way to handle syncing? I've noticed that Mail.app will reflect changes 
that I make in the gmail web interface within about 5 seconds. It can't 
possibly be checking and rechecking over 32k messages every few seconds? 
Do you know how mail clients do this? It seems like there would be a 
standard way of doing this. If not maybe I could set up my own IMAP server
and some logging and see what it's doing.
> 
> 
> 
> -- 
> Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Jan Chaloupecky
Date:
2012-07-30 @ 08:01
Nice link thanks. I'll read it through.

http://www.imapwiki.org/ClientImplementation/MessageList

The "FETCH FLAGS" would be a effective way of syncing.

The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the
entire message list.

The question is whether MailCore supports this.




On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:

> I'm checking this out to see if it has any relevant info:
>
> http://www.imapwiki.org/ClientImplementation
>
>
> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>
> I'm also not sure what is the best way to handle this. Maybe I will
> analyze the Mail.app traffic to see what it is doing.
>
> To get the new mails, I would fetch messages from local maxUID (included)
>
> To sync, I would define a user configurable option, eg "Days to Sync",
> look in local database to find the last mail matching that option, then
> fetch messages from lastMailToSyncUID.
>
>
> But you are right, this does not seem to be efficient if I want to keep
> all mail synced. It looks like we'll have to read the IMAP good practices :)
>
>
>
>
>
> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>
>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>
>> Just some quick thoughts:
>>
>>
>>>         a. call messagesFromUID and retrieve all data from the current
>>> maxUID + 1 to 0 (the last message). This will get all messages that have
>>> come in since your last update. Store them as in step #1.
>>
>>
>>
>> The UID is not sequential so the maxUID + 1 won't work I guess.
>>
>>
>> But if you give it number that isn't there won't it just give you the
>> next highest one that does exist as the first item? If not I could just
>> give it maxUID then and always throw that one out when it comes, but that
>> seems inefficient. What is the best way to say, give me all the messages
>> from this mailbox that I don't have yet?
>>
>>
>> Your algorithm will only fetch new e-mail (which are not yet stored
>> locally) but it will not handle emails that were deleted remotely.
>>
>>
>> In 2.b. when I request everything in the mailbox I could assume that any
>> message that I have already downloaded but is not in the results has been
>> deleted from the mailbox.
>>
>> All of this seems very inefficient though. Does IMAP specify a specific
>> way to handle syncing? I've noticed that Mail.app will reflect changes that
>> I make in the gmail web interface within about 5 seconds. It can't possibly
>> be checking and rechecking over 32k messages every few seconds? Do you know
>> how mail clients do this? It seems like there would be a standard way of
>> doing this. If not maybe I could set up my own IMAP server and some logging
>> and see what it's doing.
>>
>
>
>
> --
> Jan
>
>
>


-- 
Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 08:16
Also, I noticed that Mail.app doesn't always seem to be able to update 
that quickly when changes are made in the gmail. Sometimes it takes a long
time. Not sure what makes the difference.

On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> Nice link thanks. I'll read it through.
> 
> http://www.imapwiki.org/ClientImplementation/MessageList
> 
> The "FETCH FLAGS" would be a effective way of syncing.
> 
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
> 
> The question is whether MailCore supports this. 
> 
> 
> 
> 
> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
> I'm checking this out to see if it has any relevant info:
> 
> http://www.imapwiki.org/ClientImplementation
> 
> 
> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> I'm also not sure what is the best way to handle this. Maybe I will 
analyze the Mail.app traffic to see what it is doing.
>> 
>> To get the new mails, I would fetch messages from local maxUID (included)
>> 
>> To sync, I would define a user configurable option, eg "Days to Sync", 
look in local database to find the last mail matching that option, then 
fetch messages from lastMailToSyncUID.
>> 
>> 
>> But you are right, this does not seem to be efficient if I want to keep
all mail synced. It looks like we'll have to read the IMAP good practices 
:)
>> 
>> 
>> 
>> 
>> 
>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>> 
>>> Just some quick thoughts:
>>>  
>>>         a. call messagesFromUID and retrieve all data from the current
maxUID + 1 to 0 (the last message). This will get all messages that have 
come in since your last update. Store them as in step #1.
>>> 
>>> 
>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>> 
>> But if you give it number that isn't there won't it just give you the 
next highest one that does exist as the first item? If not I could just 
give it maxUID then and always throw that one out when it comes, but that 
seems inefficient. What is the best way to say, give me all the messages 
from this mailbox that I don't have yet?
>> 
>>> 
>>> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.
>> 
>> In 2.b. when I request everything in the mailbox I could assume that 
any message that I have already downloaded but is not in the results has 
been deleted from the mailbox.
>> 
>> All of this seems very inefficient though. Does IMAP specify a specific
way to handle syncing? I've noticed that Mail.app will reflect changes 
that I make in the gmail web interface within about 5 seconds. It can't 
possibly be checking and rechecking over 32k messages every few seconds? 
Do you know how mail clients do this? It seems like there would be a 
standard way of doing this. If not maybe I could set up my own IMAP server
and some logging and see what it's doing.
>> 
>> 
>> 
>> -- 
>> Jan
> 
> 
> 
> 
> -- 
> Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 08:15
I assume that is what CTFetchAttrDefaultsOnly does.

"Pass in CTFetchAttrDefaultsOnly to attrs fetch the minimum possible, this
includes the UID, RFC822.size, and flags."

So that is basically what my plan was. But I don't see it could do that 
for all messages in all mailboxes very often.

But what does the 1:68 mean? Are those the UIDs that it's retrieving? Are 
there 68 messages in the mailbox? Is it just retrieving stuff thats 
visible in the client?

Also how are you reading the IMAP commands that Mail.app is sending?


On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> Nice link thanks. I'll read it through.
> 
> http://www.imapwiki.org/ClientImplementation/MessageList
> 
> The "FETCH FLAGS" would be a effective way of syncing.
> 
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
> 
> The question is whether MailCore supports this. 
> 
> 
> 
> 
> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
> I'm checking this out to see if it has any relevant info:
> 
> http://www.imapwiki.org/ClientImplementation
> 
> 
> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> I'm also not sure what is the best way to handle this. Maybe I will 
analyze the Mail.app traffic to see what it is doing.
>> 
>> To get the new mails, I would fetch messages from local maxUID (included)
>> 
>> To sync, I would define a user configurable option, eg "Days to Sync", 
look in local database to find the last mail matching that option, then 
fetch messages from lastMailToSyncUID.
>> 
>> 
>> But you are right, this does not seem to be efficient if I want to keep
all mail synced. It looks like we'll have to read the IMAP good practices 
:)
>> 
>> 
>> 
>> 
>> 
>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>> 
>>> Just some quick thoughts:
>>>  
>>>         a. call messagesFromUID and retrieve all data from the current
maxUID + 1 to 0 (the last message). This will get all messages that have 
come in since your last update. Store them as in step #1.
>>> 
>>> 
>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>> 
>> But if you give it number that isn't there won't it just give you the 
next highest one that does exist as the first item? If not I could just 
give it maxUID then and always throw that one out when it comes, but that 
seems inefficient. What is the best way to say, give me all the messages 
from this mailbox that I don't have yet?
>> 
>>> 
>>> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.
>> 
>> In 2.b. when I request everything in the mailbox I could assume that 
any message that I have already downloaded but is not in the results has 
been deleted from the mailbox.
>> 
>> All of this seems very inefficient though. Does IMAP specify a specific
way to handle syncing? I've noticed that Mail.app will reflect changes 
that I make in the gmail web interface within about 5 seconds. It can't 
possibly be checking and rechecking over 32k messages every few seconds? 
Do you know how mail clients do this? It seems like there would be a 
standard way of doing this. If not maybe I could set up my own IMAP server
and some logging and see what it's doing.
>> 
>> 
>> 
>> -- 
>> Jan
> 
> 
> 
> 
> -- 
> Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Jan Chaloupecky
Date:
2012-07-30 @ 08:21
Yes the "1:68" means that Mail.app asked for message with UID from 1 to 68.
Again, they might not be sequential, so it does not mean there are 68
messages in the mailbox. It just means that Mail's last message has a 68
UID. I think.

I'm using Wire Shark to monitor the network connection.






On Mon, Jul 30, 2012 at 10:15 AM, Rick Gigger <rgigger@gmail.com> wrote:

> I assume that is what CTFetchAttrDefaultsOnly does.
>
> "Pass in CTFetchAttrDefaultsOnly to attrs fetch the minimum possible, this
> includes the UID, RFC822.size, and flags."
>
> So that is basically what my plan was. But I don't see it could do that
> for all messages in all mailboxes very often.
>
> But what does the 1:68 mean? Are those the UIDs that it's retrieving? Are
> there 68 messages in the mailbox? Is it just retrieving stuff thats visible
> in the client?
>
> Also how are you reading the IMAP commands that Mail.app is sending?
>
>
> On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>
> Nice link thanks. I'll read it through.
>
> http://www.imapwiki.org/ClientImplementation/MessageList
>
> The "FETCH FLAGS" would be a effective way of syncing.
>
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the
> entire message list.
>
> The question is whether MailCore supports this.
>
>
>
>
> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
>
>> I'm checking this out to see if it has any relevant info:
>>
>> http://www.imapwiki.org/ClientImplementation
>>
>>
>> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>
>> I'm also not sure what is the best way to handle this. Maybe I will
>> analyze the Mail.app traffic to see what it is doing.
>>
>> To get the new mails, I would fetch messages from local maxUID (included)
>>
>> To sync, I would define a user configurable option, eg "Days to Sync",
>> look in local database to find the last mail matching that option, then
>> fetch messages from lastMailToSyncUID.
>>
>>
>> But you are right, this does not seem to be efficient if I want to keep
>> all mail synced. It looks like we'll have to read the IMAP good practices :)
>>
>>
>>
>>
>>
>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>>
>>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>>
>>> Just some quick thoughts:
>>>
>>>
>>>>         a. call messagesFromUID and retrieve all data from the current
>>>> maxUID + 1 to 0 (the last message). This will get all messages that have
>>>> come in since your last update. Store them as in step #1.
>>>
>>>
>>>
>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>>>
>>>
>>> But if you give it number that isn't there won't it just give you the
>>> next highest one that does exist as the first item? If not I could just
>>> give it maxUID then and always throw that one out when it comes, but that
>>> seems inefficient. What is the best way to say, give me all the messages
>>> from this mailbox that I don't have yet?
>>>
>>>
>>> Your algorithm will only fetch new e-mail (which are not yet stored
>>> locally) but it will not handle emails that were deleted remotely.
>>>
>>>
>>> In 2.b. when I request everything in the mailbox I could assume that any
>>> message that I have already downloaded but is not in the results has been
>>> deleted from the mailbox.
>>>
>>> All of this seems very inefficient though. Does IMAP specify a specific
>>> way to handle syncing? I've noticed that Mail.app will reflect changes that
>>> I make in the gmail web interface within about 5 seconds. It can't possibly
>>> be checking and rechecking over 32k messages every few seconds? Do you know
>>> how mail clients do this? It seems like there would be a standard way of
>>> doing this. If not maybe I could set up my own IMAP server and some logging
>>> and see what it's doing.
>>>
>>
>>
>>
>> --
>> Jan
>>
>>
>>
>
>
> --
> Jan
>
>
>


-- 
Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 09:21
I guess this is what we really need to read:

http://www.ietf.org/rfc/rfc4549.txt

There is this small summary:

http://www.imapwiki.org/ClientImplementation/Synchronization

Do you know how to use the [account idle] command in MailCore? The API 
docs aren't too explicit. Also I don't know how reliable IDLE is on the 
gmail servers, or if they support it at all.

Here is a section from the RFC:

4.3.1.  Discovering New Messages and Changes to Old Messages

   Let <lastseenuid> represent the highest UID that the client knows
   about in this mailbox.  Since UIDs are allocated in strictly
   ascending order, this is simply the UID of the last message in the
   mailbox that the client knows about.  Let <lastseenuid+1> represent
   <lastseenuid>'s UID plus one.  Let <descriptors> represent a list
   consisting of all the FETCH data item items that the implementation
   considers part of the descriptor; at a minimum this is just the FLAGS
   data item, but it usually also includes BODYSTRUCTURE and
   RFC822.SIZE.  At this step, <descriptors> SHOULD NOT include RFC822.

   With no further information, the client can issue the following two
   commands:

      tag1 UID FETCH <lastseenuid+1>:* <descriptors>
      tag2 UID FETCH 1:<lastseenuid> FLAGS

   The first command will request some information about "new" messages
   (i.e., messages received by the server since the last
   synchronization).  It will also allow the client to build a message
   number to UID map (only for new messages).  The second command allows
   the client to

      1) update cached flags for old messages;

      2) find out which old messages got expunged; and

      3) build a mapping between message numbers and UIDs (for old
         messages).
That sounds pretty close to what I was planning on doing. I'm guessing 
though that it makes sense to not sync every single folder very often but 
to keep what the user is looking at really up to date and do every thing 
else less often. Poking around in Mail.app, seems to confirm that this is 
what it does. It doesn't even download messages from folders until you 
click on them.

There are also these extensions to help with syncing:

http://www.ietf.org/rfc/rfc5162.txt

Although, I don't know if it's worth looking into them unless the majority
of servers, or at least Google's, support them.



On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> Nice link thanks. I'll read it through.
> 
> http://www.imapwiki.org/ClientImplementation/MessageList
> 
> The "FETCH FLAGS" would be a effective way of syncing.
> 
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
> 
> The question is whether MailCore supports this. 
> 
> 
> 
> 
> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
> I'm checking this out to see if it has any relevant info:
> 
> http://www.imapwiki.org/ClientImplementation
> 
> 
> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> I'm also not sure what is the best way to handle this. Maybe I will 
analyze the Mail.app traffic to see what it is doing.
>> 
>> To get the new mails, I would fetch messages from local maxUID (included)
>> 
>> To sync, I would define a user configurable option, eg "Days to Sync", 
look in local database to find the last mail matching that option, then 
fetch messages from lastMailToSyncUID.
>> 
>> 
>> But you are right, this does not seem to be efficient if I want to keep
all mail synced. It looks like we'll have to read the IMAP good practices 
:)
>> 
>> 
>> 
>> 
>> 
>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>> 
>>> Just some quick thoughts:
>>>  
>>>         a. call messagesFromUID and retrieve all data from the current
maxUID + 1 to 0 (the last message). This will get all messages that have 
come in since your last update. Store them as in step #1.
>>> 
>>> 
>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>> 
>> But if you give it number that isn't there won't it just give you the 
next highest one that does exist as the first item? If not I could just 
give it maxUID then and always throw that one out when it comes, but that 
seems inefficient. What is the best way to say, give me all the messages 
from this mailbox that I don't have yet?
>> 
>>> 
>>> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.
>> 
>> In 2.b. when I request everything in the mailbox I could assume that 
any message that I have already downloaded but is not in the results has 
been deleted from the mailbox.
>> 
>> All of this seems very inefficient though. Does IMAP specify a specific
way to handle syncing? I've noticed that Mail.app will reflect changes 
that I make in the gmail web interface within about 5 seconds. It can't 
possibly be checking and rechecking over 32k messages every few seconds? 
Do you know how mail clients do this? It seems like there would be a 
standard way of doing this. If not maybe I could set up my own IMAP server
and some logging and see what it's doing.
>> 
>> 
>> 
>> -- 
>> Jan
> 
> 
> 
> 
> -- 
> Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Jan Chaloupecky
Date:
2012-07-30 @ 09:41
>
> tag2 UID FETCH 1:<lastseenuid+1>:* FLAGS


so you were right. The imap server will automatically get the emails that
you do not have even if the  <lastseenuid+1> UID does actually *not* exist
on the server. Nice.

I haven't tried the IDLE command yet but I will use it as a "nice to have".
Automatically syncing ever X minutes is good enough for me for the start.





On Mon, Jul 30, 2012 at 11:21 AM, Rick Gigger <rgigger@gmail.com> wrote:

> I guess this is what we really need to read:
>
> http://www.ietf.org/rfc/rfc4549.txt
>
> There is this small summary:
>
> http://www.imapwiki.org/ClientImplementation/Synchronization
>
> Do you know how to use the [account idle] command in MailCore? The API
> docs aren't too explicit. Also I don't know how reliable IDLE is on the
> gmail servers, or if they support it at all.
>
> Here is a section from the RFC:
>
> 4.3.1.  Discovering New Messages and Changes to Old Messages
>
>    Let <lastseenuid> represent the highest UID that the client knows
>    about in this mailbox.  Since UIDs are allocated in strictly
>    ascending order, this is simply the UID of the last message in the
>    mailbox that the client knows about.  Let <lastseenuid+1> represent
>    <lastseenuid>'s UID plus one.  Let <descriptors> represent a list
>    consisting of all the FETCH data item items that the implementation
>    considers part of the descriptor; at a minimum this is just the FLAGS
>    data item, but it usually also includes BODYSTRUCTURE and
>    RFC822.SIZE.  At this step, <descriptors> SHOULD NOT include RFC822.
>
>    With no further information, the client can issue the following two
>    commands:
>
>       tag1 UID FETCH <lastseenuid+1>:* <descriptors>
>       tag2 UID FETCH 1:<lastseenuid> FLAGS
>
>    The first command will request some information about "new" messages
>    (i.e., messages received by the server since the last
>    synchronization).  It will also allow the client to build a message
>    number to UID map (only for new messages).  The second command allows
>    the client to
>
>       1) update cached flags for old messages;
>
>       2) find out which old messages got expunged; and
>
>       3) build a mapping between message numbers and UIDs (for old
>          messages).
>
> That sounds pretty close to what I was planning on doing. I'm guessing
> though that it makes sense to not sync every single folder very often but
> to keep what the user is looking at really up to date and do every thing
> else less often. Poking around in Mail.app, seems to confirm that this is
> what it does. It doesn't even download messages from folders until you
> click on them.
>
> There are also these extensions to help with syncing:
>
> http://www.ietf.org/rfc/rfc5162.txt
>
> Although, I don't know if it's worth looking into them unless the majority
> of servers, or at least Google's, support them.
>
>
>
> On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>
> Nice link thanks. I'll read it through.
>
> http://www.imapwiki.org/ClientImplementation/MessageList
>
> The "FETCH FLAGS" would be a effective way of syncing.
>
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the
> entire message list.
>
> The question is whether MailCore supports this.
>
>
>
>
> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
>
>> I'm checking this out to see if it has any relevant info:
>>
>> http://www.imapwiki.org/ClientImplementation
>>
>>
>> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>
>> I'm also not sure what is the best way to handle this. Maybe I will
>> analyze the Mail.app traffic to see what it is doing.
>>
>> To get the new mails, I would fetch messages from local maxUID (included)
>>
>> To sync, I would define a user configurable option, eg "Days to Sync",
>> look in local database to find the last mail matching that option, then
>> fetch messages from lastMailToSyncUID.
>>
>>
>> But you are right, this does not seem to be efficient if I want to keep
>> all mail synced. It looks like we'll have to read the IMAP good practices :)
>>
>>
>>
>>
>>
>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>>
>>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>>
>>> Just some quick thoughts:
>>>
>>>
>>>>         a. call messagesFromUID and retrieve all data from the current
>>>> maxUID + 1 to 0 (the last message). This will get all messages that have
>>>> come in since your last update. Store them as in step #1.
>>>
>>>
>>>
>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>>>
>>>
>>> But if you give it number that isn't there won't it just give you the
>>> next highest one that does exist as the first item? If not I could just
>>> give it maxUID then and always throw that one out when it comes, but that
>>> seems inefficient. What is the best way to say, give me all the messages
>>> from this mailbox that I don't have yet?
>>>
>>>
>>> Your algorithm will only fetch new e-mail (which are not yet stored
>>> locally) but it will not handle emails that were deleted remotely.
>>>
>>>
>>> In 2.b. when I request everything in the mailbox I could assume that any
>>> message that I have already downloaded but is not in the results has been
>>> deleted from the mailbox.
>>>
>>> All of this seems very inefficient though. Does IMAP specify a specific
>>> way to handle syncing? I've noticed that Mail.app will reflect changes that
>>> I make in the gmail web interface within about 5 seconds. It can't possibly
>>> be checking and rechecking over 32k messages every few seconds? Do you know
>>> how mail clients do this? It seems like there would be a standard way of
>>> doing this. If not maybe I could set up my own IMAP server and some logging
>>> and see what it's doing.
>>>
>>
>>
>>
>> --
>> Jan
>>
>>
>>
>
>
> --
> Jan
>
>
>


-- 
Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-07-30 @ 10:05
Me too. I want to just get it syncing first. Then I can worry about making
it more efficient later.


On Jul 30, 2012, at 3:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> tag2 UID FETCH 1:<lastseenuid+1>:* FLAGS
> 
> so you were right. The imap server will automatically get the emails 
that you do not have even if the  <lastseenuid+1> UID does actually not 
exist on the server. Nice.
> 
> I haven't tried the IDLE command yet but I will use it as a "nice to 
have". Automatically syncing ever X minutes is good enough for me for the 
start.
> 
> 
> 
> 
> 
> On Mon, Jul 30, 2012 at 11:21 AM, Rick Gigger <rgigger@gmail.com> wrote:
> I guess this is what we really need to read:
> 
> http://www.ietf.org/rfc/rfc4549.txt
> 
> There is this small summary:
> 
> http://www.imapwiki.org/ClientImplementation/Synchronization
> 
> Do you know how to use the [account idle] command in MailCore? The API 
docs aren't too explicit. Also I don't know how reliable IDLE is on the 
gmail servers, or if they support it at all.
> 
> Here is a section from the RFC:
> 
> 4.3.1.  Discovering New Messages and Changes to Old Messages
> 
>    Let <lastseenuid> represent the highest UID that the client knows
>    about in this mailbox.  Since UIDs are allocated in strictly
>    ascending order, this is simply the UID of the last message in the
>    mailbox that the client knows about.  Let <lastseenuid+1> represent
>    <lastseenuid>'s UID plus one.  Let <descriptors> represent a list
>    consisting of all the FETCH data item items that the implementation
>    considers part of the descriptor; at a minimum this is just the FLAGS
>    data item, but it usually also includes BODYSTRUCTURE and
>    RFC822.SIZE.  At this step, <descriptors> SHOULD NOT include RFC822.
> 
>    With no further information, the client can issue the following two
>    commands:
> 
>       tag1 UID FETCH <lastseenuid+1>:* <descriptors>
>       tag2 UID FETCH 1:<lastseenuid> FLAGS
> 
>    The first command will request some information about "new" messages
>    (i.e., messages received by the server since the last
>    synchronization).  It will also allow the client to build a message
>    number to UID map (only for new messages).  The second command allows
>    the client to
> 
>       1) update cached flags for old messages;
> 
>       2) find out which old messages got expunged; and
> 
>       3) build a mapping between message numbers and UIDs (for old
>          messages).
> That sounds pretty close to what I was planning on doing. I'm guessing 
though that it makes sense to not sync every single folder very often but 
to keep what the user is looking at really up to date and do every thing 
else less often. Poking around in Mail.app, seems to confirm that this is 
what it does. It doesn't even download messages from folders until you 
click on them.
> 
> There are also these extensions to help with syncing:
> 
> http://www.ietf.org/rfc/rfc5162.txt
> 
> Although, I don't know if it's worth looking into them unless the 
majority of servers, or at least Google's, support them.
> 
> 
> 
> On Jul 30, 2012, at 2:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> Nice link thanks. I'll read it through.
>> 
>> http://www.imapwiki.org/ClientImplementation/MessageList
>> 
>> The "FETCH FLAGS" would be a effective way of syncing.
>> 
>> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
>> 
>> The question is whether MailCore supports this. 
>> 
>> 
>> 
>> 
>> On Mon, Jul 30, 2012 at 9:54 AM, Rick Gigger <rgigger@gmail.com> wrote:
>> I'm checking this out to see if it has any relevant info:
>> 
>> http://www.imapwiki.org/ClientImplementation
>> 
>> 
>> On Jul 30, 2012, at 1:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>> 
>>> I'm also not sure what is the best way to handle this. Maybe I will 
analyze the Mail.app traffic to see what it is doing.
>>> 
>>> To get the new mails, I would fetch messages from local maxUID (included)
>>> 
>>> To sync, I would define a user configurable option, eg "Days to Sync",
look in local database to find the last mail matching that option, then 
fetch messages from lastMailToSyncUID.
>>> 
>>> 
>>> But you are right, this does not seem to be efficient if I want to 
keep all mail synced. It looks like we'll have to read the IMAP good 
practices :)
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Jul 30, 2012 at 9:30 AM, Rick Gigger <rgigger@gmail.com> wrote:
>>> On Jul 30, 2012, at 1:16 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
>>> 
>>>> Just some quick thoughts:
>>>>  
>>>>         a. call messagesFromUID and retrieve all data from the 
current maxUID + 1 to 0 (the last message). This will get all messages 
that have come in since your last update. Store them as in step #1.
>>>> 
>>>> 
>>>> The UID is not sequential so the maxUID + 1 won't work I guess.
>>> 
>>> But if you give it number that isn't there won't it just give you the 
next highest one that does exist as the first item? If not I could just 
give it maxUID then and always throw that one out when it comes, but that 
seems inefficient. What is the best way to say, give me all the messages 
from this mailbox that I don't have yet?
>>> 
>>>> 
>>>> Your algorithm will only fetch new e-mail (which are not yet stored 
locally) but it will not handle emails that were deleted remotely.
>>> 
>>> In 2.b. when I request everything in the mailbox I could assume that 
any message that I have already downloaded but is not in the results has 
been deleted from the mailbox.
>>> 
>>> All of this seems very inefficient though. Does IMAP specify a 
specific way to handle syncing? I've noticed that Mail.app will reflect 
changes that I make in the gmail web interface within about 5 seconds. It 
can't possibly be checking and rechecking over 32k messages every few 
seconds? Do you know how mail clients do this? It seems like there would 
be a standard way of doing this. If not maybe I could set up my own IMAP 
server and some logging and see what it's doing.
>>> 
>>> 
>>> 
>>> -- 
>>> Jan
>> 
>> 
>> 
>> 
>> -- 
>> Jan
> 
> 
> 
> 
> -- 
> Jan

Re: [mailcore] syncing an IMAP mailbox

From:
Matt Ronge
Date:
2012-08-01 @ 18:18

On Jul 30, 2012, at 4:41 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> tag2 UID FETCH 1:<lastseenuid+1>:* FLAGS
> 
> so you were right. The imap server will automatically get the emails 
that you do not have even if the  <lastseenuid+1> UID does actually not 
exist on the server. Nice.

Yes you can do lastseenuid+1, in fact that is a recommended way of getting
just new messages.

Also the RFC that was linked earlier: http://www.ietf.org/rfc/rfc4549.txt 
is a gold mine of info on syncing. Study that carefuly along with the IMAP
RFC

--
Matt Ronge
mronge@mronge.com
iOS Developer & Consultant
Central Atomics Inc.

Re: [mailcore] syncing an IMAP mailbox

From:
Matt Ronge
Date:
2012-08-01 @ 18:22

On Jul 30, 2012, at 3:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:

> Nice link thanks. I'll read it through.
> 
> http://www.imapwiki.org/ClientImplementation/MessageList
> 
> The "FETCH FLAGS" would be a effective way of syncing.
> 
> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
> 
> The question is whether MailCore supports this. 

Yes MailCore does support this. As Rick said, that is what 
CTFetchAttrDefaultsOnly does. It is designed to be used for flag syncing. 
Flag syncing is generally very fast, so you can download all the message 
flags often. In fact in the base IMAP spec you have to do that to 
determine if any messages have been moved or deleted when synchronizing.

In the above case they are doing a fetch based on sequence number. If it 
was "FETCH UID 1:68" then it would be a UID based fetch. So the above 
command read "Get the flags and uid for messages numbered 1 through 68." 
Think of sequence numbers as an array index, while UID are actually unique
numbers (for that mailbox) that don't change.

--
Matt Ronge
mronge@mronge.com
iOS Developer & Consultant
Central Atomics Inc.

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-08-02 @ 00:45
On Aug 1, 2012, at 12:22 PM, Matt Ronge <mronge@mronge.com> wrote:
> 
> On Jul 30, 2012, at 3:01 AM, Jan Chaloupecky <chaljan@gmail.com> wrote:
> 
>> Nice link thanks. I'll read it through.
>> 
>> http://www.imapwiki.org/ClientImplementation/MessageList
>> 
>> The "FETCH FLAGS" would be a effective way of syncing.
>> 
>> The OSX Mail.app  seems to use "9.5 FETCH 1:68 (FLAGS UID)" to get the 
entire message list. 
>> 
>> The question is whether MailCore supports this. 
> 
> Yes MailCore does support this. As Rick said, that is what 
CTFetchAttrDefaultsOnly does. It is designed to be used for flag syncing. 
Flag syncing is generally very fast, so you can download all the message 
flags often. In fact in the base IMAP spec you have to do that to 
determine if any messages have been moved or deleted when synchronizing.

While it seems crazy to me that the way of checking for changes is to scan
the entire folder it does indeed seem to be very fast. Checking a folder 
on gmail with about 15,000 messages in it and using 
CTFetchAttrDefaultsOnly took less than 10 seconds. Using 
CTFetchAttrEnvelope on the same folder took about 40.

What kind of operations can you be notified of via IDLE? Deletes? Moves? 
New mail? Flag changes? And is there support for IDLE in MailCore or 
LibEtPan?

Rick

Re: [mailcore] syncing an IMAP mailbox

From:
Matt Ronge
Date:
2012-08-02 @ 04:19
> 
> While it seems crazy to me that the way of checking for changes is to 
scan the entire folder it does indeed seem to be very fast. Checking a 
folder on gmail with about 15,000 messages in it and using 
CTFetchAttrDefaultsOnly took less than 10 seconds. Using 
CTFetchAttrEnvelope on the same folder took about 40.

Since it is such a common operation servers are optimized for it.

> 
> What kind of operations can you be notified of via IDLE? Deletes? Moves?
New mail? Flag changes? And is there support for IDLE in MailCore or 
LibEtPan?

There is support for IDLE, except the IDLE RFC 
(http://tools.ietf.org/html/rfc2177) says "the server is now free to send 
untagged EXISTS, EXPUNGE, and other messages at any time." So basically, 
you should expect anything. That isn't very helpful (I wish the spec had 
specified what servers must send in resposne) What many clients do is go 
into IDLE mode and when any response is received do a quick fetch for new 
messages.

IDLE support can be found in CTCoreAccount
--
Matt Ronge
mronge@mronge.com
iOS Developer & Consultant
Central Atomics Inc.

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-08-02 @ 21:15
On Aug 1, 2012, at 10:19 PM, Matt Ronge <mronge@mronge.com> wrote:

>> What kind of operations can you be notified of via IDLE? Deletes? 
Moves? New mail? Flag changes? And is there support for IDLE in MailCore 
or LibEtPan?
> 
> There is support for IDLE, except the IDLE RFC 
(http://tools.ietf.org/html/rfc2177) says "the server is now free to send 
untagged EXISTS, EXPUNGE, and other messages at any time." So basically, 
you should expect anything. That isn't very helpful (I wish the spec had 
specified what servers must send in resposne) What many clients do is go 
into IDLE mode and when any response is received do a quick fetch for new 
messages.
> 
> IDLE support can be found in CTCoreAccount

I see that there is an idle function. It looks like you just call [account
idle]. Is that right? So does that just keep the connection open and 
signal to the server that we are waiting for notifications? Is there any 
example code on how to accept notifications? I guess I just need to peruse
the RFC so that I at least understand how IDLE works.


Re: [mailcore] syncing an IMAP mailbox

From:
Jonathan Willing
Date:
2012-08-02 @ 21:19
You need to call [account read]; after you call -idle. The -read command 
actually creates a blocking call which hangs there until the server 
reports changes. When that happens, the -read call is ended, and execution
continues past that line. At least, that's what I've sort of figured out 
with help from Matt. However, he said that he hasn't personally used the 
-idle support in MailCore yet, so IDLE support is still a WIP.

On Aug 2, 2012, at 4:15 PM, Rick Gigger <rgigger@gmail.com> wrote:

> On Aug 1, 2012, at 10:19 PM, Matt Ronge <mronge@mronge.com> wrote:
> 
>>> What kind of operations can you be notified of via IDLE? Deletes? 
Moves? New mail? Flag changes? And is there support for IDLE in MailCore 
or LibEtPan?
>> 
>> There is support for IDLE, except the IDLE RFC 
(http://tools.ietf.org/html/rfc2177) says "the server is now free to send 
untagged EXISTS, EXPUNGE, and other messages at any time." So basically, 
you should expect anything. That isn't very helpful (I wish the spec had 
specified what servers must send in resposne) What many clients do is go 
into IDLE mode and when any response is received do a quick fetch for new 
messages.
>> 
>> IDLE support can be found in CTCoreAccount
> 
> I see that there is an idle function. It looks like you just call 
[account idle]. Is that right? So does that just keep the connection open 
and signal to the server that we are waiting for notifications? Is there 
any example code on how to accept notifications? I guess I just need to 
peruse the RFC so that I at least understand how IDLE works.

Re: [mailcore] syncing an IMAP mailbox

From:
Rick Gigger
Date:
2012-08-02 @ 21:25
Ok. I'll try it out and post to the list if I run into any problems.

On Aug 2, 2012, at 3:19 PM, Jonathan Willing <jwilling@me.com> wrote:

> You need to call [account read]; after you call -idle. The -read command
actually creates a blocking call which hangs there until the server 
reports changes. When that happens, the -read call is ended, and execution
continues past that line. At least, that's what I've sort of figured out 
with help from Matt. However, he said that he hasn't personally used the 
-idle support in MailCore yet, so IDLE support is still a WIP.
> 
> On Aug 2, 2012, at 4:15 PM, Rick Gigger <rgigger@gmail.com> wrote:
> 
>> On Aug 1, 2012, at 10:19 PM, Matt Ronge <mronge@mronge.com> wrote:
>> 
>>>> What kind of operations can you be notified of via IDLE? Deletes? 
Moves? New mail? Flag changes? And is there support for IDLE in MailCore 
or LibEtPan?
>>> 
>>> There is support for IDLE, except the IDLE RFC 
(http://tools.ietf.org/html/rfc2177) says "the server is now free to send 
untagged EXISTS, EXPUNGE, and other messages at any time." So basically, 
you should expect anything. That isn't very helpful (I wish the spec had 
specified what servers must send in resposne) What many clients do is go 
into IDLE mode and when any response is received do a quick fetch for new 
messages.
>>> 
>>> IDLE support can be found in CTCoreAccount
>> 
>> I see that there is an idle function. It looks like you just call 
[account idle]. Is that right? So does that just keep the connection open 
and signal to the server that we are waiting for notifications? Is there 
any example code on how to accept notifications? I guess I just need to 
peruse the RFC so that I at least understand how IDLE works.
> 

Re: [mailcore] syncing an IMAP mailbox

From:
Matt Ronge
Date:
2012-08-01 @ 18:23
On Jul 30, 2012, at 2:08 AM, Rick Gigger <rgigger@gmail.com> wrote:

> I am trying to figure out the procedure for syncing an IMAP folder. This
is what I have come up with:
> 
> 1. Call messagesFromUID and retrieve all data in all messages 
(CTFetchAttrEnvelope | CTFetchAttrBodyStructure). Save it all to disk and 
record the max UID.
> 2. When you want to update
> 	a. call messagesFromUID and retrieve all data from the current maxUID +
1 to 0 (the last message). This will get all messages that have come in 
since your last update. Store them as in step #1.
> 	b. call messagesFromUID with CTFetchAttrDefaultsOnly and retrieve the 
flags for all messages. This is the only data that could have changed 
since the message was first downloaded. Compare the flags to the existing 
data and update anything that has changed.
> 
> Does that sound correct? Is there a more efficient way to do it?

You are absolutely dead on, I've implemented syncing like this myself :)
--
Matt Ronge
mronge@mronge.com
iOS Developer & Consultant
Central Atomics Inc.