Dealing with distributed transaction anomalies caused by web service calls from within transactions
I am writing a very, very, very big application at the moment and I am totally swamped in a 24/7 coding frenzy that’s going to continue for the next week or so, but here’s one little bit to think about and for which I came up with a solution. It’s actually a pretty scary problem.
Let’s say you have a transactional serviced component (or make that a transactional EJB) and you call an HTTP web service from it that forwards any information to another service. What happens if the transaction fails for any reason? You’ve just produced a phantom record. The web service on the other end should never have seen that information. In fact, that information doesn’t exist from the viewpoint of your rolled back local transaction. And of course, as of yet, there is no infrastructure in place that gives you interoperable transaction flow. And if that were the case, the other web service may not support it. What should you do? Panic?
There is help right in the platform (Enterprise Services that is). Your best friend for that sort of circumstance is System.EnterpriseServices.CompensatingResourceManager.
The use case here is to call another service to allocate some items from an inventory service. The call is strictly asynchronous and I the remote service will eventually turn around and call an action on my service (they have a “duplex” conversation using asynchronous calls going back and forth). Instead of calling the service form within my transactional method, I am deferring the call until the transaction is being resolved. Only when DTC is sure that the local transaction will go through, the web service call will be made. There is no way to guarantee that the remote call succeeds, but it does at least eliminate the very horrible side effects on overall system consistency caused by phantom calls. It is in fact quite impossible to implement “Prepare” correctly here, since the remote service may fail processing the (one-way) call on a different thread and hence I might never get a SOAP fault indicating failure. Because that’s so and because I really don’t know what the other service does, I am not writing any specific recovery code in the “Commit” phase. Instead, my local state for the conversation indicates the current progress of the interaction between the two services and logs an expiration time. Once that expiration time has passed without a response from the remote service, a watchdog will pick up the state record, create a new message for the remote service and replay the call.
For synchronous call scenarios, you could implement (not shown here) a two-step call sequence to the remote service, which the remote service needs to support, of course. In “Prepare” (or in the “normal code”) you would pass the data to the remote service and hold a session state cookie. If that call succeeds, you vote “true”. In “Commit” you would issue a call to commit that data on the remote service for this session, on “Abort” (remember that the transaction may fail for any reason outside the scope of the web service call), you will call the remote service to cancel the action and discard the data of the session. What if the network connection fails between the “Prepare” phase call and the “Commit” phase call? That’s the tricky bit. You could log the call data and retry the “Commit” call at a later time or keep retrying for a while in the “Commit” phase (which will cause the transaction to hang). There’s no really good solution for that case, unless you have transaction flow. In any event, the remote service will have to default to an “Abort” once the session times out, which is easy to do if the data is kept in a volatile session store over there. It just “forgets” it.
However, all of this is much, much better than making naïve, simple web service calls that fan out intermediate data from within transactions. Fight the phantoms.
At the call location, write the call data to the CRM transaction log using the Clerk:
AllocatedItemsMessage aim = new AllocatedItemsMessage();
aim.allocatedAllocation = <<< copy that data from elsewhere>>>
Clerk clerk = new Clerk(typeof(SiteInventoryConfirmAllocationRM),"SiteInventoryConfirmAllocationRM",CompensatorOptions.AllPhases);
SiteInventoryConfirmAllocationRM.ConfirmAllocationLogRecord rec = new RhineBooks.ClusterInventoryService.SiteInventoryConfirmAllocationRM.ConfirmAllocationLogRecord();
rec.allocatedItemsMessage = aim;
clerk.WriteLogRecord( rec.XmlSerialize() );
clerk.ForceLog();
Write a compensator that picks up the call data from the log and forwards it to the remote service. In the “Prepare” phase, the minimum work that can be done is to check whether the proxy can be constructed. You could also make sure that the call URL is valid, the server name resolves and you could even try a GET on the service’s documentation page or call a “Ping” method the remote service may provide. That all serves to verify as good as you can that the “Commit” call has a good chance of succeeding:
using System.EnterpriseServices.CompensatingResourceManager;
using …
///
///
/// activity on the site inventory service if, and only if, the local transaction
/// enlisting it is succeeding. Using the technique is a workaround for the lack
/// of transactional I/O with HTTP web services. While the compensator cannot make
/// sure that the call will succeed, it can at least guarantee that we do not produce
/// phantom calls to external services.
///
public class SiteInventoryConfirmAllocationRM : Compensator
{
private bool vote = true;
[Serializable]
public class ConfirmAllocationLogRecord
{
public SiteInventoryInquiries.AllocatedItemsMessage allocatedItemsMessage;
internal string XmlSerialize()
{
StringWriter sw = new StringWriter();
XmlSerializer xs = new XmlSerializer(typeof(ConfirmAllocationLogRecord));
xs.Serialize(sw,this);
sw.Flush();
return sw.ToString();
}
internal static ConfirmAllocationLogRecord XmlDeserialize(string s)
{
StringReader sr = new StringReader(s);
XmlSerializer xs = new XmlSerializer(typeof(ConfirmAllocationLogRecord));
return xs.Deserialize(sr) as ConfirmAllocationLogRecord;
}
}
public override bool PrepareRecord(LogRecord rec)
{
try
{
SiteInventoryInquiriesWse sii;
ConfirmAllocationLogRecord calr = ConfirmAllocationLogRecord.XmlDeserialize((string)rec.Record);
sii = InventoryInquiriesInternal.GetSiteInventoryInquiries( calr.allocatedItemsMessage.allocatedAllocation.warehouseName );
vote = sii != null;
return false;
}
catch( Exception ex )
{
ExceptionManager.Publish( ex );
vote = false;
return true;
}
}
public override bool EndPrepare()
{
return vote;
}
public override bool CommitRecord(LogRecord rec)
{
SiteInventoryInquiriesWse sii;
ConfirmAllocationLogRecord calr = ConfirmAllocationLogRecord.XmlDeserialize((string)rec.Record);
sii = InventoryInquiriesInternal.GetSiteInventoryInquiries( calr.allocatedItemsMessage.allocatedAllocation.warehouseName );
try
{
sii.ConfirmAllocation( calr.allocatedItemsMessage );
}
catch( Exception ex )
{
ExceptionManager.Publish( ex );
}
return true;
}
}