Synchronization Downloads All Data Everytime


#1

I am writing a mobile application that is updated from our CRM and allows the user to submit data to update the CRM when a job is finished. The submitted data is sent back as a “response” record and as such doesn’t update the data originally received.

To accomplish this I have a Windows Service that obtains the latest data from the CRM and updates individual user realms with this data. Another Windows Service then parses the realms looking for “unprocessed” responses which eventually will be pushed back to the CRM.

My issue is the second service originally opened the individual realm, checked for responses, then closed the realm. This didn’t work as it wasn’t allowing enough time for the realm to be synced before it was closed. I then used the following code to ensure the realm was synced before checking for the unprocessed responses;

var session = driverRealm.GetSession();
AsyncContext.Run(() => SynchroniseRealm(session));

private void SynchroniseRealm(Session session)
{
	var uploadProgress = session.GetProgressObservable(ProgressDirection.Upload, ProgressMode.ForCurrentlyOutstandingWork);
	var downloadProgress = session.GetProgressObservable(ProgressDirection.Download, ProgressMode.ForCurrentlyOutstandingWork);

	progressToken = uploadProgress.CombineLatest(downloadProgress, (upload, download) =>
	{
		return new
		{
			TotalTransferred = upload.TransferredBytes + download.TransferredBytes,
			TotalTransferable = upload.TransferableBytes + download.TransferableBytes
		};
	})
	.Throttle(TimeSpan.FromSeconds(0.1))
	.ObserveOn(SynchronizationContext.Current)
	.Subscribe(progress =>
	{
		Trace.WriteLine($"Transferred {progress.TotalTransferred} bytes of {progress.TotalTransferable}");
		if (progress.TotalTransferred >= progress.TotalTransferable) SynchronisationComplete();
	});
}

private void SynchronisationComplete()
{
	progressToken.Dispose();
}

This however appears to be downloading the entire realm every time rather than just the changes. Is there something I’m not doing correctly?

Another question I have is the trace statement is only ever executed once, apparently regardless of the size of the realm. At what point does the data become sufficiently large that progress would be reported more than once?


#2

What do you mean by “downloading the entire realm”? You appear to be listening for upload progress, not download progress. Do you mean that it’s trying to upload the entire Realm? And if so, why do you think this is the case?

Regarding the progress notifications - we bundle multiple changes in the same upload message to speed things a little, and by default the bundle size is 16 mb. We also can’t transfer partial transactions, so if you create a transaction, write a 30 mb of data, then commit it, that will be transferred as a single message. But if you have more than 16 mb of data written over multiple transactions, you should see the progress handler get called multiple times. By the way, if you’re only interested in making sure that the Realm is uploaded and don’t care too much about progress, you can just use session.WaitForUploadAsync() - it’ll simplify some of the code there.


#3

Hi Nikola,

Thanks for the explanation on the size of the download/upload packets.

I’m listening for both download and upload;

var uploadProgress = session.GetProgressObservable(ProgressDirection.Upload, ProgressMode.ForCurrentlyOutstandingWork);
	var downloadProgress = session.GetProgressObservable(ProgressDirection.Download, ProgressMode.ForCurrentlyOutstandingWork);

	progressToken = uploadProgress.CombineLatest(downloadProgress, (upload, download) =>

When the code runs it transfers that same data for each of the realms every time even though no changes have been made as the following trace statements show;

08:01:30:= Processing: Driver 1
08:01:34:= Transferred 17322 bytes of 17322
08:01:34:= Processing: Driver 2
08:01:36:= Transferred 17540 bytes of 17540
08:01:36:= Processing: Driver 3
08:01:39:= Transferred 260953 bytes of 260953
08:01:39:= Processing: Driver 4
08:01:41:= Transferred 894583 bytes of 894583
...
08:05:59:= Processing: Driver 113
08:06:01:= Transferred 296483 bytes of 296483
08:06:01:= Processing: Driver 114
08:06:04:= Transferred 660992 bytes of 660992

-----------------------------------------------------------
08:07:34:= Processing: Driver 1
08:07:36:= Transferred 17322 bytes of 17322
08:07:36:= Processing: Driver 2
08:07:38:= Transferred 17540 bytes of 17540
08:07:38:= Processing: Driver 3
08:07:40:= Transferred 260953 bytes of 260953
08:07:40:= Processing: Driver 4
08:07:42:= Transferred 894583 bytes of 894583
...
08:11:07:= Processing: Driver 113
08:11:09:= Transferred 296483 bytes of 296483
08:11:09:= Processing: Driver 114
08:11:11:= Transferred 660992 bytes of 660992

-----------------------------------------------------------
08:12:41:= Processing: Driver 1
08:12:44:= Transferred 17322 bytes of 17322
08:12:44:= Processing: Driver 2
08:12:45:= Transferred 17540 bytes of 17540
08:12:45:= Processing: Driver 3
08:12:47:= Transferred 260953 bytes of 260953
08:12:47:= Processing: Driver 4
08:12:49:= Transferred 894583 bytes of 894583
...
08:16:07:= Processing: Driver 113
08:16:09:= Transferred 296483 bytes of 296483
08:16:09:= Processing: Driver 114
08:16:11:= Transferred 660992 bytes of 660992

As you can see, every time the service runs the same data is synchronised each time. I understood that once the local realm was synchronised with the ROS then only data changes would be synchronised.

The following code is what I use to open the realms and ensure the data is synchronised;

private static readonly string appDirectory = AppDomain.CurrentDomain.BaseDirectory;

var user = AuthenticateDriver(db, driverRow);
using (var driverRealm = OpenRealm("~/RunSheet", user, driverRow.Field<String>("UserName")))
{
	var session = driverRealm.GetSession();
	AsyncContext.Run(() => SynchroniseRealm(session));
	ProcessDriverData(db, driverRealm);
}

private static Realm OpenRealm(string realmName, User user, string path)
{
	Realm realm;
	try
	{
		if (!Directory.Exists($"{appDirectory}\\Realms\\{path}")) Directory.CreateDirectory($"{appDirectory}\\Realms\\{path}");
		var config = RealmServices.GetRealmConfiguration(realmName, user, $"{appDirectory}\\Realms\\{path}", "RunSheet.realm");
		realm = RealmServices.ConnectToSyncServer(config);
		return realm;
	}
	catch (Exception ex)
	{
		Service.LogException(new ShredException(ModuleName, "Open Realm", ex.Message));
		return null;
	}
}

private static User AuthenticateDriver(SqlConnector db, DataRow row)
{
	User user = null;
	try
	{
		var driverId = row.Field<Guid>("DriverId").ToString();
		var pin = row.Field<int>("PIN");
		var userName = row.Field<string>("UserName").Trim().Replace(" ", "").Replace("'", "").ToLower();

		// Check if the driver exists in the Realm...
		user = AsyncContext.Run(() => AuthenticateRealmUser(realmInstance, userName, pin.ToString()));
		if (user == null) user = AsyncContext.Run(() => AddRealmUser(realmInstance, userName, pin));
		if (user == null) throw new Exception($"Failed to create a Realm User record for the specified driver.");
	}
	catch (Realms.Sync.Exceptions.AuthenticationException ex)
	{
		Service.LogException(new Exception(ex.Message));
	}
	catch (Exception ex)
	{
		Service.LogException(new Exception(ex.Message));
	}
	return user;
}

public static class RealmServices
{
	public static FullSyncConfiguration GetRealmConfiguration(string realmName, User user, string path, string fileName)
	{
		var serverUrl = new Uri(realmName, UriKind.Relative);
		FullSyncConfiguration config;
		config = new FullSyncConfiguration(serverUrl, user, $"{path}\\{fileName}")
		{
			ObjectClasses = new[] { typeof(Driver), typeof(Run), typeof(Job), typeof(Service), typeof(JobResponse), typeof(ServiceResponse) }
		};
		return config;
	}

	public static Realm ConnectToSyncServer(FullSyncConfiguration config)
	{
		return  Realm.GetInstance(config);
	}
}


#4

That is certainly surprising. Looking at your code, everything seems reasonable, so I don’t have a good theory as to why you’re seeing what you’re seeing. Just to confirm - is there anything in your app that may be deleting the $"{appDirectory}\\Realms folder or any files inside it? That’s the only obvious explanation, but it’s also a bit far-fetched. It would be curious to launch your app and monitor the folder between launches - it will be interesting to know if the size of the files grows and if the last modified date changes every time you launch the app.


#5

Hi Nikola,

I double checked the code for anything that would delete the realm as some of my code in the early stages would delete the realm each time however all of this code has been removed.

I then deleted all of my local and ROS realms for all drivers and ran my service that adds the data to the realm twice. The created and modified date and time for the driver realm I was monitoring was set as “13/06/2019 1:36PM” which corresponded to the date and time on the lock file. I let the service run a second time and the date and time on the realm file didn’t change but the modified date on the lock file was updated to “13/06/2019 1:46PM”.

I then ran my second service which retrieves the response records to update the CRM which is the service that is waiting to synchronise the data. I also let this run twice and the date and time on the realm file was not modifed but the date and time on the lock file was updated to “13/06/2019 1:49PM” and then “13/06/2019 2:01PM”.

Just confirming that both services access the same local realm files in case this would cause an issue although in this test the services were not run in parallel.

All of these times align to the trace statements from my code. If you want to view the logs for the service I can email them to you.


#6

Thanks for the thorough investigation. It appears that the Realms are indeed untouched, which implies that no new data has been written to them, even though the progress notifications imply that it has. I’ll try to reproduce that locally and get back to you.


#7

I was able to reproduce this and spoke with an engineer on the sync team (they own the C++ SDK that the .NET SDK uses internally to synchronize with the server) and he confirmed that this is a bug, but we don’t have a good understanding of what is causing it. He also said that the whole progress notification system is not as robust and reliable as we’d like it to be, so they do plan to rework some aspects of it.

In the meantime, if you don’t care about progress notifications, I would advise you to use the session’s WaitForUploadAsync API as that’s know to work reliably and is generally more lightweight than the progress notification one. And on a somewhat related note, I’d like to point out that we don’t support sharing synchronized realms between processes, so if you have multiple services that need to run at the same time, each needs to keep its own copy of the files.

And again, I’d like to thank you for the time you spent tracking that down and I’m sorry I wasn’t able to provide you with a fix or a better workaround for that issue.


#8

Thanks for the update. I will setup my services to use their own local realms as you suggested.

I regards to the synchronisation issue can you give me any indication when this is expected to be resolved? I can continue with the current code in the meantime using the WaitForUploadAsync and WaitForDownloadAsync methods.


#9

Unfortunately we don’t have an estimate yet on how much effort it’s going to take to rework the progress notification system. But considering the current priorities, I don’t expect it’ll happen in the next 3 months. And just to be clear, the issue is with the progress estimates, not with the underlying sync mechanisms - i.e. your service is not redownloading the Realm every time - it is indeed uploading only the changes, but the progress callback is invoked with incorrect data.


#10

Thanks for the explanation - I was concerned the data was actually being downloaded in full each time.

I have updated my code as follows;

using (var driverRealm = OpenRealm("~/RunSheet", user, userName))
{
	var session = driverRealm.GetSession();
	session.WaitForDownloadAsync();

	UpdateDriverData(db, driverRealm, driverRow);

	session.WaitForUploadAsync();
}

I am waiting for any downloads to complete as the realm may have been updated by another process. I then process the data and wait for any uploads to complete before closing the realm. Is this the right way to handle the synchronisation?


#11

You seem to be missing some await-s before session.WaitForDownloadAsync and session.WaitForUploadAsync - not sure if it’s a copy-paste error or a legitimate omission, but other than that, it seems to be correct.