Throughput II
Today was a long and tiring day, which fits in pretty well with the rest of the week. On Wednesday, we ran the new system at work, which went pretty well in the afternoon. When it got to five, we'd normally switch back to the old system for the evening shift, but we'd had no troubles all day, so called Darius to ask if he'd like to run it for the
( Read more... )
If you really want to maximize scalibility with 'built-in' reliability you should perhaps consider a message queue based system. On a *basic* level you dump messages, most likely serialized .net objects, onto queues and you if you want - receive a notification message that it has been queued. Clients then read things off the queue - with support for atomic dequeuing. It will also help if you need to scale your application up in the future as you can start distributing your queues and your processing.
You might also want to think about making use of .Net remoting to simplify things.
Cannot obviously help much with your £0 order issue but it smells of a synchronization issue on a variable that you're accessing from two separate threads.
Be careful with your enthusiasm for threading. Remember with say a quad core processor you can really only have four threads. If you start spinning off threads all over the place you're going to start running into performance problems as threads spend more time starting and stopping than they do running but I'm sure you must know this. Most server applications typically process items sequentially on a limited number of threads. The advantage is, as you've discovered, is when you start threading off blocking operations such as IO operations.
Threading is easy. It's the synchronization bit that the tricky part.
Reply
I've implemented logging of every incoming and outgoing message passing through VmspClient (and it's underlying TcpClient), and intend to run the system again tomorrow. If TCP is working like it should, I should expect to see that the outgoing logs on one side of the connection are identical to the incoming logs on the other side, and the reverse. If that is the case, the problem is entirely mine. If they don't match, some data is being sent out and never arriving, and the sender is unaware of it.
Message queueing was how I was planning to redesign the connection classes. I've already altered the server-side classes to dump incoming messages into a queue, which are then processed on another thread, and the responses placed into an outgoing queue to be send out. It seems to have helped a little, so I intend to do the same on the client side, but really I want to rewrite the lot with this design in mind from the start.
I've looked briefly at remoting, but I don't think that's what I want here. My objects are transmitted as key/value sets, which was done so that clients could request only the keys of an object that they actually need, rather than having to transmit the entire serialised object every time (some can be large). Missing keys are requested in groups of similar meanings if and when they are requested.
The £0 issue does indeed appear to be caused by two requests to manipulate the same object being processed by the server at the same time, causing unpredictable results.
As for using multiple threads for processing, currently, just for experimentation's sake, I am using the thread pool with a limit of eight worker threads. I'll play around with this to see what works best, but my main concern is treating different clients fairly. The server serves different customers of ours on completely separate accounts, and I need to ensure that one customer sending a lot of (relatively) long-running jobs doesn't cause other customers to suffer due to having to wait. I'm thinking I should maintain request queues per-account, rather than global.
It's all been interesting, anyway, I'm learning. Thanks for the feedback, I was rather surprised to receive it - I had no idea anyone with such interests was reading. Who are you?
Reply
Leave a comment