[en] Detect disconnection (solved)

Topics: Technical Support
Jul 15, 2011 at 9:34 AM

Hi again, what is the best way to detect disconnections? I got an ebc which receives events from a server, if i shutdown the server my ebc stands there as nothing ever happened...

Is there a way to figure out such problems or i must implement a polling mechanism to deal with it?

 

Thanks in advance Geo

Coordinator
Jul 15, 2011 at 3:01 PM
Edited Jul 15, 2011 at 3:03 PM

Hi, Geo!

I am not aware of any such mechanisms except for TCP keep-alives.
Standard TcpChannel enables keep-alive option with default time (2 hours, I think) and that's all.
Zyan's duplex TcpExChannel sends keep-alive every 30 seconds (you can adjust TcpKeepAliveTime option
in protocol setup classes) and tries to reconnect automatically if the connection is lost.
But it doesn't provide any event you can handle in your code to work around connection breakdown.

So, I suppose, you should implement your own polling mechanism to detect connection problems.

Coordinator
Jul 15, 2011 at 3:02 PM
Edited Jul 15, 2011 at 3:12 PM

Perhaps we could implement something like this in Zyan on a connection level, similarly to session keep-alive timer, i.e.:

var conn = new ZyanConnection(url);
conn.EnablePolling = true;
conn.PollingInterval = TimeSpan.FromMinutes(1);
conn.Disconnected += (s, e) => Console.WriteLine("Connection is lost!");

However, I'd suggest to wait a few days until Rainbird returns from his vacation (until Monday, I think).
He may have some ready-to-use recipe I am not aware of.

Jul 18, 2011 at 6:39 AM

Thanks for the info yallie.

Coordinator
Jul 18, 2011 at 8:32 PM

Hello Geo,

as yallie said, Zyan has no feature for connection loss detection, yet. It´s difficult to provide a generalized solution for this problem, because it heavily depends of the environment and the concrete scenario.
But I think it´s a good idea to extend Zyan with an easy to use connection-alive-polling feature. This is very useful for monitoring mission-critical services, too.

Unfortunately I have no ready-to-use recipe. Until the feature described under http://zyan.codeplex.com/workitem/1093 is completed, you could use a timer on client side und call a method on your server on every interval. If the connection is lost meanwhile, you get a exception, which can be catched. But keep in mind that this solution creates network traffic. So consider carefully how long your timer interval should be.

Coordinator
Aug 21, 2011 at 2:27 PM
Edited Aug 21, 2011 at 2:27 PM

Hi Geo,

Zyan has now (since ChangeSet 12893) a built-in feature for detecting unexpected disconnection.
Here is a short sample code snippet, how to use the new feature:

 

var conn = new ZyanConnection(url);
conn.PollingEnabled = true;
conn.PollingInterval = TimeSpan.FromMinutes(1);
conn.Disconnected += (s, e) => Console.WriteLine("Connection is lost! Exception details: " + e.Exception.Message);

 

There is also a possibility to make reconnection attempts. Set the Retry property of the DisconnectedEventArgs to true, if you want Zyan to retry. Retries are done as long as your set the Retry property inside your event handler to true.

For logging and troubleshooting purposes you can enable Polling Event Tracing on your host, to get notified when clients poll (sending heartbeat signals):

host.PollingEventTracingEnabled = true;
host.ClientHeartbeatReceived += (s, e) => Console.WriteLine(string.Format("{0}: Heartbeat from client session '{1}' received.", e.HeartbeatReceiveTime, e.SessionID));

Coordinator
Aug 21, 2011 at 10:27 PM

Hi Rainbird, that's great!

What about the client's event handlers?
When the server fires some event while the client is disconnected, network exception is thrown, and subscription gets cancelled.
Can we have some safe way for automatic re-subscription?

// Consider the following client setup code:
var conn = new ZyanConnection(url);
var server = conn.CreateProxy<IMySingletonServer>();
server.RegularServerEvent += (s, e) => Console.WriteLine("Server event has happened."); // subscribe

Then, after some time:
1. Connection gets broken.
2. Server singleton component fires client's event handler and catches NetworkException.
3. Server cancels the subscription.
4. Connection is restored.
5. Client don't receive notifications anymore.

 

Coordinator
Aug 22, 2011 at 7:03 PM

Hi yallie,

At the moment, the lifetime of remote event and delegate handlers isn´t bound to the session lifetime. This is also a security issue. A client can receive callbacks, even if it is logged off already.
If we bind the event and delegate handler lifetime to the session lifetime, orphaned handlers will be cleaned up, when the session gets cleaned up. So temporary disconnected clients have time to reconnect until the session expires.
But behavior of ZyanDispatcher then must be changed. It shouldn´t remove handlers of unresponding clients, because there is a chance that they may reconnect.

But this will slow down event handling (Depending on the used protocol and config, many seconds). A quick solution is to call every single client handler in a seperate thread. The executing worker thread, which fired the event on serverside, should be blocked until all handler calls are processed (e.G. with ManualResetEvent). The number of parallel executed handler calls is limited by ThreadPool settings (20 by default).

An additional solution may be using the heartbeat signals (only possible if when polling is activated). ZyanDispatcher can suspend unresponding handlers and resume them, if their session gets a hearbeat. This prevents the dispatcher from dumb and expensive connection attempts.

ZyanComponentHost may express callback errors in a special event to notify the server environment that handler X for event Y in session Z couldn´t called successfully. In case of the suspend/resume solution, resuming an handler can also be expressed in this way.

Coordinator
Aug 24, 2011 at 9:28 AM

>At the moment, the lifetime of remote event and delegate handlers isn't bound
>to the session lifetime. This is also a security issue.
>A client can receive callbacks, even if it is logged off already.

It seems to be easy to fix.

I can add SessionID to DynamicEventWireBase and check SessionManager.ExistSession
before actual invocation. If the session doesn't exist, event handler is removed.

>temporary disconnected clients have time to reconnect until the session expires.
>But behavior of ZyanDispatcher then must be changed. It shouldn't remove handlers
>of unresponding clients, because there is a chance that they may reconnect.

That's not exactly what was in my mind.

I was thinking of re-subscription (client calls AddEventHandler again for every
event handler upon reconnection). Although it probably doesn't look nice :)

>An additional solution may be using the heartbeat signals (only possible if when
>polling is activated). ZyanDispatcher can suspend unresponding handlers and resume them,
>if their session gets a hearbeat. This prevents the dispatcher from dumb and
>expensive connection attempts.

That's a good idea!

We can resume suspended handlers on remote calls as well.
If the session gets either a heartbeat or remote call, then handler is re-activated.

>ZyanComponentHost may express callback errors in a special event to notify the
>server environment that handler X for event Y in session Z couldn't called successfully.
>In case of the suspend/resume solution, resuming an handler can also be expressed in this way.

Diagnostic events are always a good idea :)

Coordinator
Aug 25, 2011 at 8:09 PM

> I can add SessionID to DynamicEventWireBase and check SessionManager.ExistSession
> before actual invocation. If the session doesn't exist, event handler is removed.

Exactly.

> I was thinking of re-subscription (client calls AddEventHandler again for every
> event handler upon reconnection). Although it probably doesn't look nice :)

But if events not fired while disconnected, they are still registered. Calling AddHandler will register them again.
When the event then is fired after reconnection, the client handler will called twice.
Did you thought about that?

> We can resume suspended handlers on remote calls as well.
> If the session gets either a heartbeat or remote call, then handler is re-activated.

Yes, then it will also work without polling enabled.

Coordinator
Aug 25, 2011 at 11:26 PM

>But if events not fired while disconnected, they are still registered. Calling AddHandler will register them again.
>When the event then is fired after reconnection, the client handler will called twice.
>Did you thought about that?

Yes, I did. I just assumed that it's easy to work around.
Correlation info is identified by a Guid, which could be attached to a DynamicEventWire.
When a proxy calls AddEventHandler() for the same CorrelationInfo, the dispatcher can skip duplicate subscription.
I didn't dive deep into details, but basically something like that :)

>Yes, then it will also work without polling enabled.

Yes, certainly.

Coordinator
Aug 26, 2011 at 4:52 AM

We should implement both reconnection strategies.