Lynn's Industrial Protocols over IP

This blog has moved

2010-04-06T07:16:00.001-07:00

This blog is now located at http://iatip.blogspot.com/.
You will be automatically redirected in 30 seconds, or you may click here.

For feed subscribers, please update your feed subscriptions to
http://iatip.blogspot.com/feeds/posts/default.

How Long to Obtain Cellular IP?

2010-04-06T06:39:00.000-07:00

Well, I finally have a SIM, so I can actually do cellular tests again. It is a GSM 'Capped Plan' with 20/MB of data per month for $30. The 'capped' means that even if I go over the 20MB by dozens of GB, the maximum charge will be $130. It is a truly 'unlimited plan' for that $130. What's the catch? If I hit the 'Cap' two months in a row, then the carrier either cancels the account or forces me to start paying full-rate for a normal 1GB or 5GB/mo plan.

The 20MB is fine for me since I am testing methods to keep data small.

My current tests have a sleep devices waking, then obtaining a cell-link. For example, I am working on a Digi gateway called an X3 (not on Digi's web site yet). The X3 can power itself down completely, using a real-time clock chip to wake it at a future time.

I ran a test for 5 days, waking the X3 device every 10 minutes and clocked how long it too for the GSM carrier to assign an IP to the X3:

The fastest time was 23.7 seconds (half a minute)
The slowest time was 97.0 seconds (1 and a half minutes)
The average time was 28.6 seconds (so most near 30 seconds, with a few long outliers)
Note that there will be several seconds of CPU start-up time in there - I can only know the RTC wake-up time (which I loaded into the RTC last cycle) compared to the RTC time when the IP address is assigned.

Why is this important? Because the cellular modem consumes a LOT of power for the duration of this variable time. It is one of the variable which makes estimating battery life difficult.

My next test will include actually doing work - probably sending a Modbus/TCP transaction to a DSL-based server.

Cellular DNS Lookup Cost

2009-10-13T14:09:00.000-07:00

I have been working with a customer using two cellular devices with a DDNS provider, so the 'Client/Master' device used a DNS name to connect to the remote 'Server/Slave' device.

Cost #1 - added delay
The first cost they were seeing was added time lag to open a connection.

To open a TCP socket between two idle (but connected) cellular devices requires each device to be "unparked" by their cell tower. So the client device would need to request a resource and be assigned it by the local cell tower - this can require 2 to 5 seconds.
Next, a DNS query would be sent and a response would be required. Some cellular carrier DNS servers don't seem very peppy, so this could add a few more seconds.
Then the client device would receive notice from the cell-tower control channel to request a resource and be "unparked"
Finally, after "X" seconds of delay and lag the client device would receive confirmation that the socket is open (or it might have given up to fast already & aborted).

So using a DNS name instead of an IP Address might add a few seconds delay - which might be just enough to cause problems. The customer I am working with has a client tool with a hard-coded MAX timeout of 25 seconds, and it seems to often hit this during original socket open. So they have no trouble communicating once the TCP socket is open, but it can be difficult to get the first data packets moving.

Cost #2 - added traffic
Of course the DNS query also adds raw traffic. DNS uses UDP, and a query might be 44 bytes and a response 60 bytes. So 104 bytes extra to open a TCP socket (which can be hundreds of bytes by itself).

Is this traffic important? Perhaps. Perhaps Not. Most cell plans today round up traffic hourly, so an extra 104 bytes for DNS and 150 bytes for a Modbus poll/response is still "1K of traffic".

For TCP/IP, the DNS lookup would only occur for a socket open. So regardless of the DNS name Time-To-Live, you might only see this added 104 bytes once per day or month - or anytime you open/reopen a socket.

However for UDP/IP you MIGHT see this DNS lookup for every packet. For example, a DDNS record from dyndns.org might expire in 60 seconds, so sending one 100 byte UDP packet every 5 minutes, could become 204 bytes per 5 minutes, so 2448 bytes per hour rounded up to 3K per hour or 72K per day and NOT the 49K expected.

So while UDP/IP can save a lot of traffic cost over TCP/IP, if short-lived DNS names are required then some of UDP's savings will be lost.

One solution is a custom application to do indirect DNS lookup. In other words, to directly lookup the DNS name within the application, then to ignore the DNS "Time-to-Live" and use only the cached IP address UNTIL COMMUNICATIONS FAILS - then do a new lookup. This is not how normal operating system DNS caches work - they literally honor the DNS server's time to live and would cause a new look up based on the 60 second time-to-live. Yet this is general safe for cellular end-devices since normally the dynamic IP only changes when the cell link is lost, so the remote IP is unlikely to change more than a few times per day.

Does Cellular 3G help your Rockwell PLC Access?

2009-06-10T06:33:00.000-07:00

In short - No.

Cellular data systems are continuously being sub-optimized for the unidirectional fire-hose paradigm of web page viewing, media streaming, image viewing, email download and so on. In other words, you only see the 3G "broadband" performance when you have tens of thousands of bytes to shove in one direction.

Actual field tests of RSLogix 5000 downloading to ControlLogix processors show that using 3G is only a few percent faster than standard technology - plus still probably slower than analog dialup. The problem is caused by RSLogix and Ethernet/IP moving thousands of small transactions in a half-duplex manner. So RSLogix downloads the PLC object by object, confirming success after each object is written. With cellular latency, each of these transactions can take up to a second to complete, so multiple 1 second by 3000 to 8000 transactions and you have your PLC download time.

3G apologists will claim "Oh, but latency in 3G is much lower than older, coarser technologies."

That may be, but when the rubber meets the road (aka RSLogix5000 downloads), 3G doesn't perform any better. 3G might have other value, but for most people moving data "faster" just means your data usage and the final bill racks up faster as well.

Cellular to Redundant Rockwell ControlLogix

2009-06-06T06:32:00.000-07:00

Can RSLinx access a redundant pair via cellular?

Background
A Pair of Rockwell Automation ControlLogix racks with SRM Module and dual ENBT's will share a pair of IP addresses. One IP address is "the primary", and the other IP is "the backup" (if/when it is online). When the ENBTs switch role, they will issue the requisite Gratuitous ARPs to cause other local Ethernet devices (like a Digi cellular gateway) to update their ARP cache, thus comprehending that the IP-to-MAC address mapping has changed. Thus a NAT/Router forwarding to the primary IP should handle the fail-over with only modest bumps.

Config Details
A user desiring RSLinx (and RSLogix or RSView etc) to access a remote Rockwell ControlLogix (any RA/AB PLC) will be doing what the industry calls "Mobile-Terminated" access. The user needs to arrange a cell plan which offers either a fixed IP address to target, or at least a Dynamic DNS name to target (like tk101.iatips.com or panel2.digi.com). This is NOT what you obtain with an iPhone or personal air-card data plan. Those will have private IPs which only permit outgoing connections - called "Mobile-Originated". That was a buzzword lesson - expect to be asked about those two terms when you ask about cellular data plans!

So once you can arrange your targetable IP or DNS name, you need a cellular router such as one of the Digi Connect WAN family products. My favorite model today is the ConnectPort X4, but that has large memory for Python programming, wireless mesh and other goodies you won't need to link up your Rockwell PLC (Hey, I said it was MY favorite - doesn't mean it has to be yours!)

Note that contrary to folklore or urban legend, all cellular devices need certification to work on a system - even GSM devices. Many small suppliers get around this by including fine-print that say the device buyer is responsible to arrange such legalities, and since you (the buyer) don't read such fine print the salesperson will just say "Heck, it's GSM - so it is allowed everywhere world wide!" Deal with this issue as you see fit, but Digi has more formal certs in more countries worldwide than any of the other industrial players.

But back on subject, when the gateway comes up, it is assigned your known IP or DNS name. This is exactly how your home or business DSL/T-line line works. Yet when RSLinx tries to talk to the gateway on Ethernet/IP's well-known TCP port of 44818, the gateway will reject the connection as a weird attempt at hacking. You need to instruct the gateway:

to not reject the Ethernet/IP traffic on TCP port 44818

to instead forward it to a local IP on the Ethernet - which would be the IP of your ControlLogix ENBT (or the primary IP of the redundant pair)

The details of how this man-in-the-middle fake-out works is fascinating (to me), but quite a pile of text. If you are interested, this older blog entry goes through the NAT/Router details blow by blow. But bottom-line, your ENBT receives the RSLinx packet and needs to have its own Gateway IP set to the Digi gateway's local Ethernet IP address. Free hint: 9 out of 10 guys who call saying "Why can't RSLinx see my PLC through your Digi gateway?" have failed to set the correct Gateway IP in the PLC/ENBT.

So assuming your gateway and PLC are setup correctly, then targeting the RSLinx "Ethernet Devices" driver (with timeouts slowed down to 30 seconds) will cause your PLC's little icon to show up. With RSLinx running, you will create up to 200MB of billable cell traffic per month doing absolutely nothing - so don't leave it active. Note that the "Ethernet/IP Driver" won't work as it requires UDP broadcast, which can't be routed over the Internet.

At this point you'll say "Cool, now can I see my backup ControlLogix or a second PLC?", and the simple answer is "No." One of the realities of RSLinx and AB PLC is that the Ethernet/IP protocol is hard fixed to only the TCP/IP port 44818, and the NAT/Router can only forward TCP port 44818 to a single local PLC. The easy fix would be for Rockwell to change RSLinx to enable adding both an IP/DNS name and TCP port number - then the NAT/Router could forward TCP port 44818 as 44818 to the primary ENBT, TCP 44819 fixed-up to 44818 to the secondary ENBT, TCP 17256 (a random number :-]) fixed up to 44818 to an RSView panel and so on. Because the NAT/Router can restore all traffic to 44818 on the local Ethernet, RSLinx is the only tool needing to change.

Will Rockwell ever do this? It would take a programmer half a day to do - then a few weeks to test - then a few months to document and forestall support headaches. So who knows. They might. They might not.

But bottomline is a simple cellular NAT/Router can be used to talk to a pair of ControlLogix running in a redundant configuration - you will just be limited to seeing only the primary ENBT and the primary IP address.

Cellular Antenna and your PLC

2009-04-07T05:56:00.000-07:00

One of my customers is having fun with one of their own customers. My customer uses Digi gateways running a Python application to collect hourly tank levels, which are fetched by cellular once per day. The tanks hold 250-gallons of a chemical additive which is injected at a variable rate into crude oil pipelines. Small battery-powered ultrasonic level sensors (www.massa.com) push the last 8 hourly samples every few hours using wireless IEEE 802.15.4 (aka Zigbee). The end customer's goal is to forecast when the tanks need to be refilled ... and I don't mean just receive an alarm when it is running low, they literally want to plan truck routes in advance.

Fun stuff.

The problem is that one of the pilot sites has bad cell coverage, meaning many days the central system cannot upload the log. Of course the data eventually it all uploads since it is held for over a month. Now, never mind that this system sits down in a dry wash (a small valley), the end-user says "Hah, that's because carrier A*** stinks; we all love carrier S****." My customer does not want to mix carriers - especially since negotiated pricing between the two in question is so different.

So remember that "coverage" for fixed RTU must be viewed differently from "coverage" for mobile uses.

For a mobile device (or user) "coverage" is defined by the probability that a valid carrier connection will be available as the device moves around. Thus the number of towers (etc) is important, and if there is no signal in one spot, then hopefully will be one a mile or two away.

Unfortunately, our little RTU bolted to a power pole in that gully won't be moving a mile or two ever. So in the end, "coverage" for a fixed device is defined only by the tower(s) which can be seen. So the carrier with the best cell-phone coverage might not be the best carrier to support a particular fixed RTU.

The first step will be to move the RTU up out of the gully, which is easy since the data signals are all wireless and power can be tapped from any of the privately owned power poles. Probably carrier A*** (stinky or not) will work fine once the RTU moves. As plan B and make the end-customer feel listened to, a S****-based RTU will be placed on the same pole as the relocated A*** one.

Could they run out a long external antenna? Sure, they could. But why bother? Put the RTU where it has a nice signal. Even in 2 or 3 store buildings we suggest customers put the cellular router up in the roof-area and run the Ethernet UTP it's 100 meters instead of trying to deal with the signal loss in long antenna cables. We even have customers using 900Mhz Ethernet bridges to link a cellular router placed where it must be placed back to the Ethernet devices which want access.

Worst case, a directional yagi antenna could be used, however you need to understand that cell towers are routinely turned off without warning. The carriers are truly geared towards mobile users who expect bad signals in some places. Towers (or the active elements) also move. Most of the cell towers you see along the highway work like strip-malls; a company owning the tower and supplying power leases tower space to a mix of carriers. This allows everyone to be flexible.

So prematurely locking down a cellular device to a single tower with a directional antenna can cause future problems since it will not see other weaker towers should the targeted tower be turned off or even moved.

Ethernet/IP PCCC Service Codes

2008-11-25T06:29:00.000-08:00

Summary: the Rockwell PCCC Service Codes (or how to get a 1761-NET-ENI or Digi One IAP to send specific DF1 DST/SRC bytes)

A customer had a product which acted as an Ethernet/IP originator (client) and needed to talk through a Digi One IAP to feed DF1 into a DH+ gateway. The catch is the DH+ gateway used the DF1 destination byte as the DH+ node address - and so far they could only make the Digi One IAP spit out a destination = 1. This made for pretty boring and unprofitable DH+ design!

(Sorry: I cannot explain "How to write an Ethernet/IP stack" in a single blog entry, so this entry assumes you already have a product speaking Ethernet/IP and just want to tweak how you talk to remote DF1 devices.)

The PCCC object (code: hex 0x67) is not part of the ODVA spec - it is a vendor specific object used by Rockwell/Allen-Bradley to talk to SLC5, PLC5E and MicroLogix PLC. I won't fully explain it here ... not much to explain; do a wireshark trace of a ControlLogix talking to a SLC5/05 or MicroLogix 1100 and you'll see all you need to see about the simple object.

The default service code of 0x4B has this form:

Exec PCCC Service	4B
IOI to PCCC Object	02 20 67 24 01
Originator Info	07 03 85 50 4d 41 41
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

The only mystery here should be the "Originator Info". The 07 means a total of 7 bytes (including the 07). 0385 is a unique sequence number which you should change between messages. The 504d4141 is a "CIP Serial Number" which must be unique to your vendor id - most vendors just use the last 4 octets of an Ethernet MAC address.

However, there is NO remote node or slave address info, so the Digi One IAP or 1761-NET-ENI will create a DF1 message for destination=1 and source=0. So back to the customer's question ... how to force a DF1 message with say destination=5 and source=2?

The easiest solution is to switch to service code 0x4C, which has this form:

DH+Like Service	4C
IOI to PCCC Object	02 20 67 24 01
DH+ Like Header	00 00 02 00 00 00 05 00
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

So the "Originator Info" has been swapped with an 8 byte structure of the form AA AA BB XX CC CC DD XX. The "XX" are control bytes of some sort - just leave 0x00. "AA AA" is the Destination Link; "BB" is the Destination node; "CC CC" is the Source Link; "DD" is the Source node.

Voila - both the Digi One IAP and the Rockwell 1761-NET-ENI will now create a DF1 message with destination=5 and source=2. Not too painful, was it.

To my knowledge all Rockwell Ethernet/IP PLC accept either code, and if you'd like to see the service 0x4C in action just set your RSLogix5000 MSG block to use "CIP with Source ID" and the ControlLogix switches from using service 0x4B to using 0x4C.

For completeness, there is a final service code of 0x4D with this form:

Local PCCC Service	4D
IOI to PCCC Object	02 20 67 24 01
Example PCCC Message	0f 00 5c 00 a2 14 07 89 00 00

It is documented in DeviceNet gateway manuals to reduce total byte count - but it also lacks any destination or source info so is of limited use in Ethernet-based systems.

Creeping Firewalls and Change

2008-08-07T07:05:00.000-07:00

Summary: Network infrastructure changes can create unexpected havoc (surprised?)

We had a customer come in with a complaint recently about some old products - a 4-port terminal server (a TS4) running Modbus bridging firmware. Things had gone on smoothly for years, but suddenly they started having problems which power-cycling the units solved ... obviously it's a Digi problem, right?

Well, I'm not going to run through all of the issues, but I'll highlight one which will affect more and more people over time. Remember the old days - first you had dumb dialup modems, then you had to pay a lot for various error-detection standards, and now all analog modems have a big chip which does a hundred different standard things and you CANNOT buy any low-cost analog dial-up modem which is any dumber than state-of-the-art.

Such creep occurs in Ethernet switches also - once hubs were cheap and switches cost a king's ransom. Now even the "cheap" $19 home switches purchased in big-box stores are auto-sense, auto-cross, auto-everything. The same things happens - once the features are rolled into software or firmware, future revs of the product inherit such features for free ... said another way, it is often cheaper to sell lots of one powerful model than handle the marketing and support costs of a dozen models of various power and feature-set.

Well, this customer had been upgrading their wide-area-network infrastructure - a tangle of routed IP-based leased lines, old analog lines, radio/microwave links and so one, and as part of the improvement the various router/PPP end-points had added stateful awareness of packets ... part of an increase in US government security demands for utilities. After all, when you have hundreds of raw Ethernet ports scattered across rural American protected with little more than a padlock and chain-link fence, one has to consider the possibility of someone trying to 'jack' back into the 100% private network from a remote location.

The issue with such an addition is that routers with firewalls commonly default to a 5-minute TCP life rule. Every stateful firewall creates a table entry for every TCP or UDP activity going "out" from a safer network and into a wilder, less safe network - your home DSL/Cable router does the same thing. NAT (etc) is optional, but bottom line is the firewall needs to understand and agree to the context of every packet trying to pass through.

For a TCP socket this usually means at least 1 packet (data or ACK or Keepalive) must move every 5 minutes to refresh the context and inform the firewall that the TCP socket is still authorized. If no packets are seen, the fire wall discards the context and the socket is now ignored and 100% blocked without comment. Of course both TCP peers might think the socket is alive and well, but they will never be able to use it again.

So back to our TS4 off in some remote field - a host computer has 4 TCP sockets open to it through various routed subnets and for whatever reason the host doesn't send any new packets for 6 minutes. The TS4 has no reason to talk since it is connected to Modbus slaves. So 6 minutes later the host issues 4 Modbus requests on 4 sockets, which hit the stateful firewall. The firewall looks up the context (called forwarding or trigger rules in some systems) and discovers there is no existing context for these TCP packets. Well, once upon a time the firewall would just assume since these were safe since they came from the safe side, and thus auto-create a new context, yet modern stateful firewalls might NOT do this since it was a common hacker exploit to fool firmwalls to move malformed packets. Thus modern security demands that the first packet of any new TCP socket be a [SYN] packet. All four packets are silently discarded as unknown (note: this is a configurable behavior in most firewalls - to auto-recreate a context for an 'old' TCP socket continuing or to only allow this for new TCP sockets.)

The TS4 never sees the four new request; the host sees neither an [ACK] nor a [RST] of the TCP socket, so it retries after 3 seconds, then 6 and so on. Eventually it comprehends the TCP socket has died and opens four new TCP sockets. These are allowed through the firewall since they contain the correct TCP state as being new sockets. The TS4 sees the four new socket requests and now has eight (8) sockets connected.

Now you might say "What a minute - the four old sockets were closed!". Well, yes - the HOST knows they are dead and closed them, but the TS4 does not know this. Six minutes later, the same is repeated and the TS4 now has 12, then 16, then 20 sockets open. Eventually the TS4 runs out of resources, and the customer's open recovery option is a hard reboot of the TS4. This is of course worse if the host only occasionally takes longer than 5 minutes to poll, since the increase in TS4 sockets could take an unpredictable amount of time to build up.

How to solve this problem?

The first line of defense in the modern IP age is to always, ALWAYS enable TCP keepalives with an aggressive 4 minute 30 second idle time in any product you install. Even if you don't have stateful packet inspection within your IP network today, that doesn't mean the next time someone replaces a router or serial PPP end-point you won't gain one. Unfortunately the Digi products usually default to a 2 hour TCP Keepalive - which is also disabled (for historical reasons). Yet having a 4.5 minute keepalive would have saved the TS4 because the first time the host delayed longer than 4.5 minutes to poll, the TS4 would have sent a TCP keepalive packet back through the stateful firewall, which the host returns and the firewall's context for this socket is refreshed. Thus when the host does finally send the next Modbus poll after 6 minutes (1.5 minutes after the TCP keepalive exchange) the firewall is satisfied with the packets and forwards them out to the TS4. Everyone is happy. Plus even if the TCP socket has been aborted by the firewall, the TS4's TCP keepalive will be silently discarded and the TS4 will retry and eventually comprehend that the TCP socket is no longer valid. This is the purpose of TCP Keepalive - to allow a TCP peer with no data to move to test (and refresh) the health of an idle connection.
The second possibility is unique to the Digi IA/Modbus application. You can enable a setting referred to as "idletimeout" on incoming client or outgoing server connections. Unlike the TCP Keepalive which create traffic and keeps an idle socket open, the idletimeout literally just aborts an idle socket without giving it any chance to prove it is healthy. So setting a 5 minute idle timeout in the TS4 would cause it to just assume any incoming Modbus client (master) connection which has NOT sent a new request is bad. This setting would also have saved the TS4, forcing the four old sockets to be aborted before the TS4 had a chance to build up a herd of dead sockets.

So understanding TCP Keepalive is a critical skill for all modern industrial Ethernet users, after all more and more facilities are breaking their floors and functional area into distinct subnets with stateful firewall protection to keep the Accountants from changing the color of the widgets being produced - and to keep the production engineers from printing on the accounting departments laser printer which uses red ink :-).

More complete discussion of TCP Keepalive is here (one of dozens of web site you can find in a web search).

Tunneling Serial Data over Cellular

2008-08-01T09:51:00.000-07:00

Tunneling Serial Data over Cellular

Question: you have a Windows-based application which expects to talk over normal serial ports - how can you connect to a remote serial device over a cellular-IP link?

Products: works with Digi Connect WAN, WANIA or WANVPN, Digi ConnectPort VPN, X4 or X8

Answer: Our Digi RealPort driver for Windows 2000/XP/Vista dated Nov 2008 now supports a nice low-overhead "UDP: Serial Data Only" mode. This will cost a fraction of what normal TCP-mode Realport or competitors TCP redirectors will cost.

Here is a web page explaining how to install RealPort for UDP mode (DDNS names can be used!)
http://digitips.wikispaces.com/Digi+Realport+with+UDP

Here is a web page explaining how to set the WANIA to UDP Sockets.
http://digitips.wikispaces.com/Digi+UDP+Sockets+WAN

I set this up yesterday and have been polling a Modbus/ASCII serial slave on my CPX4 using RealPort and UDP mode. Responses take from 2 to 3 seconds to return … in part because I am polling so slowly that the modem PARKS in between every poll and I wait 30 seconds for a response timeout. I see perhaps 0.25% error / no response timeout but then my SIM doesn’t have the best signal where my product sits.

However, I still believe people need to be realistic – even in the situations where the slave response times out I’d suggest NOT retrying immediately since the retries have a higher than normal probability of failing as well. Missing a few polls a day is better than needlessly paying more for data plans.

Evolution of Data Plan Billing

2008-07-23T12:28:00.000-07:00

Summary: the big three have moved away from unlimited data, towards limited data.

It is interesting - I once (as in last year) had a talk with a potential partner who'd been at some European conference and was convinced the world was on the verge of low-cost (sub-$20/month) unlimited cellular data plans. We were discussing the creation of report-by-exception tools to reduce SCADA costs, and this partner's strong faith in this belief caused them to eventually bail out of the talks, saying "In a year or two, no SCADA company will care about how much cellular data they use."

Yet as of the summer of 2008 the world of cellular data is moving in the opposite direction. Last year the big three (AT&T/Sprint/Verizon) offered "Unlimited Data" for personal users with the Service Terms listing a VERY narrow list of permitted activities - mainly email and web browsing, with many common things like file download/upload, media-streaming prohibited. So when ever one of the big three would cut off a user for moving too much data on an "unlimited plan", the service provider would fall back on the "You are doing prohibitted things, thus impacting our network, thus take your business elsewhere". What a way to cause bad feelings, eh? Note that this change is CONSUMER plans - machine-to-machine have always been limited, priced by the MB/month without rollover, plus with charges for data overages.

Now all three have dropped the price from the $80/month range down to $60/month range ... but added a hard limit of 5GB per month. Isn't free & vigorous market competition wonderful?

Sounds reasonable - 5,000 megabytes of data is a lot, yet this doesn't mean 5GB of data transfer. It means 5GB of metered activity, with many activities I've studied including up to 95% overhead. Thus someone only moving 20-30MB of real data in small packets per month might hit pretty close to their 5GB limit! My experience with normal wide-area-network traffic hints that a real PC user doing simple email and web-browsing once a day would probably move 1-2GB of data before hitting the 5GB total activity limit.

To paraphrase the wireless data service terms for all three:

Data transport is always measured in full kilobytes
Actual transport is always rounded up to next full-kilobyte at "end of session"
Network overhead and resend requests caused by network errors can increase measured kilobytes.
2 of 3 mention always rounding up to nearest kilobyte every hour period.
All warn that you will NOT receive an itemized detail of how your charges are calculated; you will NOT see which services were used or during which time periods the charges were inccurred under.

So if I send a single 50 byte UDP/IP packet, is that a full session and billed as 1024 bytes? Could be under this language since UDP is 'sessionless'.

Hmm, the term session is pretty ambiguous. Perhaps it means per "time you enable your PC-based cellular data card." That seems likely - plus if you left your device on twenty-four hours a day then the once per hour round-up would catch you.

I'm afraid I haven't offered any new answer here, other than to suggest you understand that low-cost unlimited data plans ARE NOT just around the corner ... at best we left them behind last year and I don't foresee them ever returning. I suppose all three now understand that huge new profits are to be made with these 5GB limits, which will cause many "super-salesman" using their cellular data plan daily to spend an extra $50 to $500 in monthly overage charges.

Lower Cost Cellular to Rockwell AB PLC

2008-07-11T09:56:00.000-07:00

I have several customers now working through how to manage cost-effective cellular access to Rockwell PLC such as ControlLogix, CompactLogix, Micrologix 1100 and so on. Unfortunately the most straight forward way to link using Ethernet/IP is fairly costly.

First, a personal recommendation from me – a free tool which I find very useful and think you will too. Today, you can buy 2GB USB flash drives for $15 – if you're old like me, you remember when an entire Windows computer only had 0.020GB of hard drive space! Did you know you can literally install and run many Windows applications from these portable USB drives? This means any Windows computer you plug this USB drive into has your applications, your settings, and your data files. I've used one of these for over a year and it is invaluable - all free open source code too! You can run OpenOffice (which can read/write MSOffice 2003 files and is much faster than the MSOffice 2007 we use at Digi), Firefox web browser, plus a dozen other tools. Take a special look at KeePass, which I use daily from my USB drive to securely hold all of my hundred-plus account names and passwords.

Portableapps.com - What is a Portable App?

Okay, back to work.

Periodic PLC Access from RSLogix

Customers who want to peek into a single PLC at a remote site for an hour or two can use RSLinx to connect to either an IP or DNS name, then see the PLC via cellular. The catch is RSLinx will create from 12MB to 200MB of background traffic per month. So you need to create a new Ethernet Driver (not Ethernet/IP!) JUST for this one-time use, configure in your details, connect and do your work. When you are done you need to turn browsing off, then delete the comm driver. Why not just delete the IP or DNS name? Unfortunately once RSLinx has seen a device, it can be like a bad rash to get rid of it.

Data polling – at Central Office

Customers with an OPC server speaking DF1, CSPv4, or even Ethernet/IP can poll PCCC-type data through a Digi One IAP, which converts the polls into DF1 Radio Modem under UDP. Tests have show using DF1 Radio Modem every few minutes accomplishes the same data movement as Ethernet/IP with only 5% the data cost (or Ethernet/IP uses 2000% more data bytes). One unit of Digi One IAP can poll up to 60 remote IP or DNS names. If your OPC server can encapsulate DF1 Radio Modem directly into UDP/IP, then you won't need the Digi One IAP to act as your host.

Data Polling – at Remote Site

If you have an AB PLC which speaks DF1 Radio Modem directly, then any Digi cellular router can be configured for UDP Sockets, with shuttles UDP data received to the serial port. Make sure you use the latest Digi firmware so it can just return UDP responses to last sender without explicit address configuration.

If your AB PLC doesn't speak DF1 Radio Modem, or you want to use an Ethernet link, then using a Digi cellular router with Python support allows a simple script which accepts DF1 requests and uses a local Ethernet/IP session to query responses from the PLC's PCCC Object. This Python code even runs on a PC under Windows or Linux. As soon as I have a link or web page explaining how to get and use this code, I'll edit this post to add it here.

Quick Data Comms to AB PLC

2008-07-10T14:54:00.000-07:00

One of my readers was asking for a quick way to talk via Ethernet to a Rockwell AB PLC.

You can actually talk to a ControlLogix by only understanding TWO (2) different packets, each with a response, so four packets I guess. The problem is this uses "UCMM" style communications, which the PLC has very limited resources for. Said another way, the CIP Connected Messaging or I/O production both include an inherent allocation of resources, while the UCMM is designed to be used ONLY to setup such pre-allocated resources.

So yes, you can use the information below to create a literal quick-n-dirty solution, and if you talk more than a few times a second you might start to interfer with other communications to the PLC (which is not a good thing!) However you could treat this as a proof of concept, and then work to do the communications more fully per the ODVA specs.

Here is the PDF of how to read/write to the PCCC Object in a Logix Processor here.

Public Internet Risk in Common Tools

2008-05-30T07:06:00.001-07:00

Two months ago a SCADA customer asked me to enable FTP (File-Transfer-Protocol) on a test RTU they'd sent me to put online for them. It was on a DSL link and although I warned them it was a bad idea they said it would be okay because the RTU had username/password protection and the RTU had nothing important on it.

The punchline is that a few days later the customer sent me an email saying they couldn't FTP into the RTU anymore, so couldn't check the log files.

I looked, and the RTU now had 3 TCP sockets open (all the sockets allocated for FTP on this RTU) to some FTP client at an IP address registered in Korea. All 3 were just slowly walking through a dictionary attack of username/passwords (user bill, pass honda14 ... user billk, pass 12tomes ...) No doubt the IP and FTP client belonged to some university student running kiddy-scripts obtained on the Internet. No doubt the kid probably didn't even care if the odd device he or she had never seen before was not a computer (FTP servers always announce what and who they are when you connect). No doubt this attack wasn't costing them anything, as either the university or parents were paying for the Internet connection - or the IP and connection belonged to some patsy whose home computer had been compromised.

So okay, it was not causing any real harm, except it defeated the purpose of enabling FTP since the end user no longer had access to FTP. I suppose you could call it a denial-of-service attack, yet I'm sure that was NOT the intension of the 'attacker'. The student was probably just hoping to be able to post a message on some forum saying 'I hacked a computer at this IP address in the USA, and here is the FTP user name and password I created for you to access.' The fact that the RTU only contained a dozen binary log files would be irrelevant.

Do I have a moral to this story? Hmm, not really - other than industrial users have to understand that what is NOT interesting to them might be interesting to others for very different reasons.

If this user had allowed me to change the FTP port to some random value like TCP port 38207, then it is very likely this particular student would NOT have found it. Since true port-scans are so easy to detect, the student's tool probably had a list of a few dozen TCP ports commonly used by FTP servers, then it would randomly try them over a few weeks at any single target IP.

The same story could be true for web servers on TCP port 80 (or 8000 or 8080). Digi ships our cellular products with the web server enabled on port 80 because that is what customers expect. Sure, they add a username and password, but what will happen to their data plan bill if their 3MB per month plan moves 3-GigaByte because some scripts are trying dictionary attacks on the login of home web page?

I've had to always mention that 3GB part (meaning a $500+ bill for a month) since every time I mention to such users not to leave port 80 setup as a web browser, the industrial customer's answer is invariably '... it'll be okay because the unit has username/password protection and it has nothing worth trying to see on it ... '

Cellular to Wireless Zigbee

2008-05-22T09:34:00.000-07:00

An interesting new market we are moving into is cellular access to wireless mesh (Zigbee as example). In some sense it's a supply-chain dream come true. Imagine you supply a product to customers and EVERYONE (except the customer's IT department) want you to be able to see what the stock level is and auto-schedule deliveries.

The customer benefits because they can treat the product as a 'utility' - turn the tap and there it is.

You benefit because you can minimize emergency truck-rolls - no more angry, panicked customers demanding you send a truck over two-thirds empty because someone forgot to schedule a special delivery because the customer needed to use 60% more product for two days. Of course as a supplier the cost of the truck, fuel, and driver are critical parts of your margin/profit. You desire to only send out full trucks which return gracefully empty!

So we are now working with several of the largest chemical suppliers in the world to enable:

drop in a powered cellular unit at the customer site
drop in powered or battery tank sensors
log levels hourly, for the supplier to upload daily (reduces cellular data charges) The supplier uses this as their 'secret-sauce', their own proprietary value-add to predict when trucks need to roll to maximize efficiency
enable alarm call-out if the levels hit unexpected low-low levels

How this works varies by suppliers. The one I'm working with is using Modbus/TCP to pull up the logs daily. Some other suppliers are having SMTP clients push emails back to the supplier with XML formatted reports. The next supplier I might work will wants the binary logs to be compressed (ZIP'd) and then pushed upstream once a day by FTP, where their accounting system will convert from binary to XML to import and issue bills on product usage per MINUTE.

Of course key to all of this is the wireless drop-in-network concept. The supplier doesn't want to invest thousands of dollars pulling wires through SOMEONE ELSE'S PLANT - especially when the supplier's contract might end in a few months.

Wireless sensors aren't new; cellular data access isn't new; supply-chain systems which auto-detect product levels aren't new. What is new here is the merger of many technologies which reduce infra-structure costs, and thus increase ROI.

PCCC Protected Typed Logical Write with Mask

2008-03-14T07:42:00.000-07:00

Someone asked about the "DF1 Supplement for SLC500" from 1995 which was online at ab.com for a short period, then was pulled - probably because it is a very poor quality optical scan. However, I had it ... then lost it ... then found it stashed away on one of my ftp sites.

So here is the original AB PDF (which I downloaded from ab.com last year) Grab it here while it survives.

The masked write is on page 11 and the first data-word is the mask and all data following has the SAME mask applied. So [0x0001,0x0000] would clear the LSBit and so on. It doesn't offer mask-data pairs - just one mask and N words. Thus this command is mainly for use with 1 word element writes ... unless you have three or four consecutive N-file words you wish having the same mask applied.

Optimizing Modbus for Cellular

2008-03-07T09:27:00.000-08:00

Goal: lower your data costs

How nice it would be if you could take your Ethernet applications and just move them to cellular (or satellite). Well, of course you can ... but you'll pay through the nose for this.

At the moment I'm in the process of creating intelligent cellular gateways accessible by Modbus/UDP (aka Modbus/TCP form in UDP) which support data logging, report-by-exception and other cost-saving goodies.

A few Facts:

You are charged for all IP, TCP, and UDP overhead - the cellular system moves your TCP/IP packet as raw payload encapsulated in mobile-IP or other transports. So to them the 40-52 bytes TCP/IP header as NOT DISTINGUISHED from your data. ( My Blog entry on this )
Thus Modbus/UDP (aka Modbus/TCP in UDP/IP) will save you from 60 to 90% of your data costs. It is a single one-shot request followed by a single one-shot response. In contrast TCP/IP might require up to 400 bytes of socket open & close overhead, plus TCP acknowledge packets.

So if you are concerned about cost, you should first make sure your DATA POLLING can be done with Modbus/UDP - not TCP. No sweat if you need to use Modbus/TCP to reprogram or monitor your RTU or PLC short-term, just make sure your 24/7 repetitive data polling in via UDP. ( Is UDP reliable enough? )

So I've defined a few extensions (read as 'heresy' to many). Full Details are Here at iatips.com:

I allow use of the full Modbus/TCP header - so one can read 500 registers in a single request. This greatly reduces charges for header overhead
I allow returns LESS than data than requested when the context is appropriate. This saves having to pay for data padding, plus not having to poll a status register to see how much logged data is waiting
I allow use of data compression like ZLIB. Bottomline, 'ZIP' compression of small data sucks, but since I can return 500 registers (1K bytes) ZLIB starts showing value.
I allow packing multiple Modbus-ADU below a single header, which (unlike pipelining) signals the gateway to return multiple responses in a single packet.
I am looking at support for simple AES encryption, not as true 'security', but as a good-enough means to support Modbus without making development difficult.

If you are interested in the details, I have more discussion on my wiki site.

Lost My SIM

2008-03-07T09:21:00.000-08:00

Well, I've been quiet for awhile - lost my "free development SIM" as part of Cingular's reorg into AT&T. Thus my collection of PLC with free demo access are off-line. Is interesting to support cellular without a SIM ... not that my boss is ignoring this, but there are 'plans in the works' which involve several companies ... plans which just keep moving out.

However, I am working on Cellular gateways with intelligence now. Goal is to allow a Modbus client/master to come in once per day and upload time-stamped logs; plus if certain events occur use Modbus to call for help.

How Exposed in Public Cellular IP

2007-10-05T10:28:00.000-07:00

One of the concerns about having a PUBLIC IP address for cellular is your exposure to public hackers probing public IPs for services. In theory, you pay for all of these attempts since the mobile IP system encapsulates and transports them all on your behalf.

I'm happy to report that after logging such attempts for many months my cellular devices receive less than 4 probes in any one day and likely under 20 total per month. So many days pass with no probes at all and it appears to stay under 2-3K per month. This compares to perhaps 200 probes per day on my DSL router/firewall.

I am not sure why the difference, although I'd guess it has to do with the high initial latency in contacting a cellular IP. So any "script-kiddie" tool scanning IP address ranges probably is not willing to wait up to 5 seconds for cellular devices on busy towers which need "unparking" to respond.

What kinds of probes are they? Mainly those looking for MS-SQL servers, with a rare access to FTP and the remainder of accesses aimed to seemingly random, unnamed ports - likely associated with trojans or zombie networks.

The Truth about Cellular IP

2007-08-22T06:19:00.000-07:00

I have been very busy analyzing real-world telemetry traffic over cellular IP, and unfortunately I am now 100% convinced that you cannot effectively use most (any?) off-the-shelf "Ethernet" software tool to talk to remote Ethernet devices over cellular IP. Bottom-line is that - unless your host app is custom written to be data cost and time delay sensitive - your data costs will be bloated due to the nature of the tool. Even something as "obvious" as adding compression doesn't solve the problem because telemetry packets tend to be too small for effective compression. For example: an 8-byte Modbus/RTU request becomes 12-bytes after ZIP-style compression. Plus this doesn't reduce the 104-bytes of TCP overhead nor 28-bytes of UDP overhead. None of the cellular providers allow use of RFC-class TCP header compression, since it requires all of the infra-structure to maintain
copies of headers etc.

So I have been working on "reduction" solutions - how to obtain the effect of moving "X" IP packets but only moving "X-minus-a-bunch" of actual IP packets.

Tunneling TCP thru UDP
The most promising and generic form of reduction is to tunnel TCP/IP via UDP/IP over cellular. So the host application talks TCP/IP to a local proxy, which acts as the TCP end-point. All of the TCP SYN, ACK and Keepalive traffic is limited to the local Ethernet. The local proxy then initiates a UDP "session" with a remote proxy over cellular & we instantly see a 60-90% reduction in data costs. The remote proxy initiates a TCP/IP connection to the remote Ethernet device, which again isolated the extra TCP overhead to the remote Ethernet.

The reaction of non-IA network engineers to this idea is predictable and a bit humorous after a while. They immediately say "You cannot do that!!! UDP/IP is unreliable!!! You'll break something!!! You are committing a mortal Sin!!!" But in reality none of the IA protocols leverage the reliability of TCP anyway. For example, Rockwell RSLogix doesn't send a program block to a ControlLogix and blindly assume it was successful after the TCP Acknowledge from the peer is processed. Instead, RSLogix sits (blocks literally) and waits for a successful CIP response on a single CIP Connection. So if the local proxy returns a TCP-ACK to the RSLogix host and the CIP request is lost within the UDP/IP tunnel ... eventually RSLogix times out the CIP connection and the application (and/or user) will restart.

Fortunately, cellular is very reliable - all of my tests sending 10,000 UDP packets rarely even lost 1 packet and I'm not sure if such a rare loss is due to cellular or just my test script hiccupping & dropping a packet. Plus cellular tends to have only very bursty error problems. In other words, you won't lose 1 packet per 10,000; instead you'll lose all packets for 5 minutes or just 5 random packets out of a group of 10 sent. This shotgun-damage tends to confuse TCP/IP state machines to the point that they abort the connection anyway. In truth, in all of my Wireshark/Ethereal trace reviewing I have never seen a single situation where a TCP retry did anything but add data cost; every TCP retransmission just results in a "Duplicate ACK" showing up a few packets below in the trace & a doubling of the cost of that block of data.

So overall, anyone planning to use cellular should first investigate if they can use UDP/IP instead of TCP/IP.

TCP Problem #1 - added cost for pointless ACK
As mentioned above, real-world analysis of telemetry use of TCP shows the TCP ACK isn't useful; but worse, Embedded TCP devices tend to sub-optimize the ACK timing to "speed up" data transmission and recovery. Almost universally moving an IA protocol via TCP/IP results in 4 TCP packets instead of the idealized 3.

Your app sends a TCP request (request data size + 40-52 bytes of overhead)
800-1100 msec later your app receives a TCP ACK without data (another 40-52 bytes of overhead)
10-40 msec later your app receives the protocol response (response data size + 40-52 bytes of overhead)
Within a few msec, your app sends the TCP ACK without data.

So what would have been a 2-packet transaction with only 56 bytes of overhead under UDP/IP, or what should have been a 3-packet transaction with 120-156 bytes of overhead under ideal TCP/IP usually becomes a 4-packet transaction with 160-208 bytes of overhead. Yes, there exists a TCP socket option and the concept of "Delayed ACK Timer" to prevent the first empty TCP ack from being returned over cellular, but few embedded products use this since it adds code complexity, and it slows down overall data communications. At least in the IA world it seems everyone wants their Ethernet Product costing 2-4 times more than their serial product to appear lightning fast. So they ignore the TCP community's decades of hard-earned experience and "hack" their TCP stack to sub-optimize fast local Ethernet performance.

So this is where the instant 60-90% data cost savings of using UDP over TCP comes from. UDP has smaller headers and results in fewer packets being sent. Since the cellular IP system is "encapsulating" your TCP/IP packets in a manner similar to PPP, the entire IP header, TCP or UDP header, and your data is all considered billable payload.

There is also a myth propagated to this day that the TCP ack causes retry to occur more rapidly out in the wide-area-network infrastructure. The rhetoric goes, "If the 3rd and 4th router link is congested and the TCP data packet is lost, then the 3rd router will retransmit ... which is faster ..." Perhaps this was true back in the 1980's, but today the 3rd and 4th router (and all of the other 20 to 30 routers in a cellular end-to-end path) are just tossing IP packets upstream with no awareness of the packet functions. In reality, it is only the TCP state machines within your host machine and within your remote device that have any ability to retransmit anything.

TCP Problem #2 - added cost for premature retry
The TCP RFC includes many dynamic timers that automatically adjust themselves based on real-world performance. This is actually pretty neat. It means if the TCP ACK and response times tend to be longer than normal, then the TCP state-machine slowly increases the delay before retransmission. But I've seen 3 problems with this.

The most effective way to leverage auto-adjust is to include the 12-byte TCP header options that time-stamps all packets. Linux system add this by default and installing one of many PLC engineering tools on your Windows computers causes Windows to also start always using this. The setting generally is global - you either have 40-byte TCP headers or 52-byte TCP headers forever. So for small telemetry packets, this adds a disproportionately large increase in data costs.
Many embedded devices (PLC, RTU and I/O devices) have "hacked" the TCP ACK sub-system to force connection failure to be faster than the standard 3-4 minutes. For example, I worked with one large PLC company which expected TCP sockets failure in less than 1 second, so they forced TCP retransmission in hundreds of msec and without any normal exponential backoff between retries. This is totally unusable over cellular; you will end up with 30% to 90% of your data traffic being premature retries and responses to premature retries. I have literally seen Wireshark/Ethereal traces which are mainly black lines with red text - which is the default color used to show TCP "problems" such as lost-frag, retrans, dup-ack, etc.
The latency in cellular is abnormal by an order of magnitude. Even browsing the internet or doing a telemetry polling test over DSL/cable broadband averages latencies in the 100-150 msec range. This is what a Windows or Linux defines as "slow/bad" - not the 800 to 3500msec of cellular. So even watching a Windows or Linux TCP state machine auto-adjust the retransmission delay over time, you will not see it achieve a 100% effective setting which eliminates wasted TCP retransmissions. The delay seems to top out
at about 1.5 to 1.8 seconds, which is just too close to the actual "normal" latency range. So again, use of UDP/IP frees the use user from data costs associated with TCP legacy assumptions - both the main-stream MIS/IT market variety of assumptions and the misapplied IA vendors "speed-ups".

TCP Problem #3 - uncontrollable SYN/Socket Opens

Given the way all cellular systems "park" inactive cellular data devices, it is exceedingly rare to ever see a host app open a new TCP socket without prematurely retrying/retransmitting the SYN packet. This is because one is virtually guaranteed that it will take about 2.5 seconds for the data device to be given active airwave resources and return the SYN+ACK response. This has NOTHING to do with the "always connected" feature Digi and others claim. The data device (even when parked) is fully connected by IP and fully authenticated by the system - it is "always connected". However, the local cell tower only has finite airwave resources, so any device (cell phone or data device) which is idle from 3 to 45 seconds is "parked" without having any preallocated airwave resources. Literally when the TCP "SYN" shows up, the cell tower has to use the control channel to inform the data device to request airwave resources, and after these are requested and allocated the data device can receive and response to the TCP socket open request.

But that's not the real problem related to TCP Socket Opens ... the real problem is yet another case of IA vendors sub-optimizing TCP behavior for fast local Ethernet performance. For example, I once had a customer who normally paid about $40 per month receive a $2000 bill one month. It turns out they had powered down the remote site for 3+ days and the off-the-shelf 3rd-party host application they used would try to reopen the TCP socket every 5 seconds!!! So Windows would send the initial TCP SYN to start the open, since the remote was off-line Windows would retransmit this TCP SYN a few seconds later. After a total of 5 seconds, the application would ABORT this TCP socket attempt and start a new one. So this host app was pushing 24 billable TCP packets per minute out to a remote site that was powered down. This was nothing the host app vendor documented, nor was it anything a user could configure or over-ride. The user could configure the host app to ONLY poll once per 5 minutes; but the user had no control over this run-away TCP SYN/Open behavior.

Tunneling TCP through UDP effectively decouples the TCP SYN/Open from cellular data charges. The first TCP Syn/Open request to the local proxy would succeed even if the remote IP site is offline. No retries would be required. Even if the host app attempts to retry the data poll every 5 seconds, this is something the UDP proxy can be configured to "resist". If the user truly wants data packets to only move every few minutes, that is something the UDP proxy can easily enforce.

TCP Problem #4 - sub-optimized TCP keepalive

The final problem I'll discuss (but not by any means the "last" problem with TCP) is that many embedded IA devices have relatively fast TCP Keepalives hard-coded to speed up lost-socket detection. While this is an admirable goal, a Rockwell PLC sending out a TCP Keepalive at a fixed 45 second interval can create up to 6MB of monthly traffic by doing this. Siemens S7 PLC seem to issue TCP keepalive every 60 seconds - a bit better, but not by much. Maybe such a heart-beat is useful to know the remote is accessible, but given the reliability of cell phones (when the last time you had a dropped call or no signal ...) you'll obtain a lot of false-alarms if you treat every missed packets as something requiring maintenance's attention.

Again, tunneling TCP through UDP effectively eliminates the automatic, possibly uncontrollable use of TCP Keepalive. If your process can handle you talking to it once an hour, then the cost of TCP socket open and close, as well as any TCP Keepalive is all wasted investment.

Not only this, but the cellular providers do NOT want users who send a simple, rather empty packet every 30 to 60 seconds - this is literally the worst kind of customer, as this forces the cell tower to "waste" one of its very limited airwave resources with almost no income returned to the carrier. From what I hear, carriers either want customers who talk constantly and pay huge monthly fees (say $90 to $350/month); or they want customers who rare talk and pay a small fee (even just $5/mo) but cost the carrier virtually no direct expenses.

Putting this is "restaurant terms":

A cellular data device that talks constantly but pays for a large plan is like a restaurant patron who sits at a table, constantly ordering more food and paying a larger bill.
A cellular data device that rarely talks is like the restaurant patron who comes in once a month, sits at a table, orders a meal, pays and then vacates the table.
A cellular data device that keeps an idle channel open full time but rarely talks is like the restuarant patron who sits at a table in the resturant, reading the paper but rarely ordering food or paying a bill.

In fact, in private chats with carrier account people, I have heard several times that they have been directly to prefer either customers who talk constantly on large plans or those who talk at most once an hour (better once a day) on small plans. Customers planning to talk every few minutes have been defined as bad investments. It may be fair to say that after years of building up the data-plan customer base, the cellular carriers have come to understand that the REAL cost of data plans is not the bulk data bytes moved; it is instead the percentage of time the device consumes (or squats on) 1-of-N scare airwave resources in proportion to the monthly fee they pay.

Report-By-Exception? Maybe not

2007-07-06T07:55:00.000-07:00

One of the PLC vendors I am just starting to deal with was bragging about the power of their report by expection abilities ... how wonderful for lower cellular data costs they claim. Unfortunately they only support TCP/IP - no UDP/IP. I'll cover this application (& name the PLC) more after I have some time to work at it.

However, the big problem with their "report-by-expection" is that the PLC holds the TCP/IP socket open 24/7 and sends a TCP Keepalive every 1 minute. Sending a TCP Keepalive (2 x TCP packet headers with no data) every 60 seconds costs about 3.4MB per month! I have a hard time seeing that as "low cost". They would be better to just poll real data every 4 minutes and eliminate the TCP keepalives!

Hopefully there is a setting in the PLC to change this behavior, which is why I need to look into this more. I just want to point out that just because a system claims to only move data during exceptions ... that does NOT mean it has very low cellular data costs. Don't forget that data costs include things you don't normally see - TCP socket open/close, TCP keepalive, TCP ack, and TCP retransmissions are all things you pay for.

Real World Cellular - ControlLogix PLC

2007-06-12T11:43:00.000-07:00

Summary: Before I listed some real world numbers for Modbus polling. This time I walk through some of the costs and issues of using ODVA Ethernet/IP to talk to a Rockwell ControlLogix PLC.

The Convoluted Path of Wide-Area-Networks:
In general the magic of IP hides reality from us all. We tend to think "now I am browsing Google.com or iatips.com", but we don't really understand how COMPLEX and MIRACULOUS this really is. Your computer is NOT connected to either of these web servers; instead your computer uses the services of a dozen or more other computers/routers to get from "here" to "there". Every single data byte must be forwarded hop-by-hop through all of these cooperative peers.

As example, here is a Trace Route (tracert) of access from a computer within my test lab to a ControlLogix PLC sitting six (6) feet away. I am using public Internet access via a cellular Digi Connect WAN to the Ethernet (ENB) of the ControlLogix. Some of the public IP have "X" entered replacing the digits; you don't need to really know the exact IP value.

My computer has private IP = 10.9.92.1
01 01 ms 10.9.1.1 (Digi's private Intranet)
02 01 ms 10.10.11.10 (Digi's private Intranet)
03 01 ms 10.254.254.2 (Digi's private Intranet)
04 16 ms 66.77.x.x (Digi Co-Host/Internet Link)
05 04 ms 69.8.x.x (Digi Co-Host/Internet Link)
06 64 ms 66.77.x.x (Digi Co-Host/Internet Link)
07 09 ms min-core-02.inet.qwest.net [205.171.128.110]
08 11 ms cer-core-02.inet.qwest.net [67.14.8.18]
09 12 ms cer-brdr-01.inet.qwest.net [205.171.139.62]
10 39 ms qwest-gw.cgcil.ip.att.net [192.205.32.97]
11 35 ms tbr2.cgcil.ip.att.net [12.123.4.254]
12 35 ms tbr2.sl9mo.ip.att.net [12.122.10.46]
13 75 ms tbr2.attga.ip.att.net [12.122.10.137]
14 31 ms 12.122.85.157
15 34 ms 12.86.140.146
16 * Request timed out. (Part of Cellular Infra-Structure)
17 * Request timed out.
18 * Request timed out.
19 * Request timed out.
20 1276 ms mobile-166-XXX-XXX-XXX.mycingular.net [166.XXX.XXX.XXX]
Digi Connect WAN has private local IP = 192.168.196.80 (is 'gateway')
ControlLogix PLC has private local IP = 192.168.196.21

These traces always amaze me - how something so seemingly trivial takes so much effort to really function. Notice how my lab PC has to route through 6 devices to even get out of Digi's company network, then through Qwest (our ISP), through AT&T (my cellular SIM provider), through some unnamed hops of the cell system, and finally be port forwarded to the ControlLogix PLC. The packets may be passing through Minneapolis, Chicago, Detroit, Atlanta, and then finally returning to the PLC sitting right beside me.

Effect of NAT (Network Address Translation)
Now lets look at what happens when RSLinx on my PC opens an ODVA Ethernet/IP socket to the ControlLogix PLC. Every TCP/IP packet requires 4 unique values which define a connection:

Destination IP (target device)
Destination Port (target application within device)
Source IP (return address to originator)
Source Port (likely random port, originator is waiting for responses here)

So we start out with the 4-tuple DST=166.x.x.x : 44818 and SRC=10.9.92.1 : 22256. The 166.x.x.x IP is assigned by my cellular carrier. Port 44818 is ODVA's "well-known" port for Ethernet/IP. 10.9.92.1 is an internal Digi selected private IP. TCP port 22256 is the ephemeral (or random) port selected by RSLinx to listen for responses.

The first NAT effect is the Digi corporate firewall changes the request to be DST=166.x.x.x : 44818 and SRC=66.77.x.x : 22256. My private IP of 10.9.92.1 is meaningless out in the Qwest or AT&T's networks, so something needs to swap this for a "real" world-unique IP leased by Digi. Our corporate NAT interface creates a record (with a lifetime of 5 minutes) that allows any responses to be correctly restored to 10.9.92.1

The second NAT effect is when the Digi Connect WAN forwards to the ControlLogix with another private IP. So the 4-tuple now becomes DST=192.168.196.21 : 44818 and SRC=66.77.x.x : 22256. The ControlLogix thinks IP host 66.77.something is connected to it - not the real host IP of 10.9.92.1. Plus the ControlLogix has NOT CLUE that the RSLinx thinks the ControlLogix as IP of 166.something.

Now, to send a response the ControlLogix issues a TCP/IP packet with the flipped 4-tuple of DST=66.77.x.x : 22256 and SRC=192.168.196.21 : 44818. The Digi Connect WAN restores (undoes) the NAT and changes this to DST=66.77.x.x : 22256 and SRC=166.x.x.x : 44818. After passing back through AT&T and Qwest, Digi's corporate NAT interface restores its own NAT and changes it back to DST=10.9.92.1 : 22256 and SRC=166.x.x.x : 44818.

This understanding of NAT and IP is useful for understanding the capability and limitations of cellular access to certain devices with certain protocols. A future entry will cover setting up RSLinx Classic and using RSLogix 5000 to download over cellular to a L5555 processor.

Unlimited Cellular Data - Revisted

2007-05-31T12:23:00.000-07:00

As I go around dealing with large utility customers and hear feedback from Digi salespeople who survive large accounts, I hear some interesting but disturbing trends rising.

First, there are the people (usually who are new to cellular) who claim any day now the age of cheap, unlimited cellular data plans means all my hard work to understand or offer reduced traffic are wasted effort. I especially hear this coming from European partners.

Then there are the other people ... people I know work with very large, very powerful end users who fail to get the cellular plans they desire. Things I hear:

I have heard of customers who pilot projects and hear promises of unlimited traffic, but when the time comes to sign the contract for 3000+ sites, the carrier decides that they cannot offer unlimited traffic ... period. Hmm, I guess this is the difference showing between the carrier's commissioned sales staff and the business managers who need to keep the cellular system profitable.
I have worked with large customers who do manage to get "unlimited deals" for a modest sized system - say 50 or 100 sites, but the carrier insists on adding the 2 clauses 1) the carrier has the right to artificially slow down the data communications (without detailing what this means) and 2) the carrier has the right to just stop all the customer's data traffic temporarily if the cell system gets busy (again, not details of this defined). Hmm, I have to kind of wonder what kind of control engineer agrees to such "clauses"? You'd think trying to get one's data under control to avoid the need for unlimited data is a wiser design.
I have heard that overall the cellular carriers are starting to rethink the value of machine-to-machine data plans. Unlike DSL or cable, this is NOT an issue of bandwidth; it is an issue of the % of the time the device "hogs" 1 of N slots or resources on the tower forever. Imagine having 10 or 20 such devices squatting there, locking up that tower resource. So it is not even so much an issue of talking once per few minutes verse flat out unlimited traffic. In either case 1 of N finite tower resources is used, so long-term the only "good" data plan may be for a data system using the cellular resource every few hours or a few times a day.

So overall it looks like my efforts to understand and reduce traffic is useful.

I will give Webinar 31-May-2007

2007-05-29T12:00:00.000-07:00

31-May-2007 (Thursday) I'll be giving a webinar related to outdoor use of IP-to-serial devices. While not directly related to my blog theme of "Industrial Protocols over IP", remote devices frequently are in bad locations, where extended temperature ranges, conformal coating, and Class 1 Div 2 certs are desired.

Title: Connect Local and Remote Devices in Hazardous Environment SCADA Applications
Register here: WebEx Link

Date: May 31, 2007
Time: 11:00am CDT (central zone)
Duration: 1 hour
Speakers:

Lynn Linse - Engineer, Digi International
Deb Smith - Business Development, Digi International

What you will learn
This live webinar will illustrate the value of using TCP/IP & UDP/IP based communication over wired and wireless networks to monitor and manage local and remote devices. Topics include:

Overview of requirements for robust, remote, outdoor communication devices
Discussion of extended temperature, conformal coating, hazardous certs
Comparing features of IT-grade network devices to Digi's Haz product line
Extending IP networks to remote serial and Ethernet devices
Moving serial communications through TCP/IP and UDP/IP
Migrating from analog Telco/POTS lines to IP-based broadband
Real-world usage examples
Implementation benefits
- Real-time access to remote sites and process data without site visits
- Reduce repair/replacement costs due to less-than ideal environments
- Options to reduce installation costs with wireless

Please register today and join us May 31st for this informative webinar!
Questions? Contact Deb Smith at deb_smith@digi.com or 952-912-3283.

ODVA, Rockwell, and Modbus Get Smart

2007-05-04T12:53:00.000-07:00

Summary: next week an amazing joining of technologies will take place at ODVA in Ann Arbor MI as ODVA, Rockwell, and Schneider-Electric discuss how to integrate Modbus servers (slaves) into CIP Producer/Consumer systems

One of the fun things about being involved in "multi-vendor" solutions is when you recognize moments of amazing sanity as they occur. One such moment of amazing sanity is occurring next week when ODVA (aka Rockwell / Allen-Bradley) and Modbus supporters (aka Schneider-Electric / Modicon / SquareD / Telemecanique) sit down to discuss how to integrate Modbus devices into the ODVA Ethernet/IP and CIP network systems. Of course there must be some interesting hidden politics behind this move - and I somewhat light-heartedly believe that perhaps French Schneider-Electric sees joining with the Americans (Rockwell/ODVA) as the lesser of two evils when compared to joining with the Germans (Siemens/PNO).

Check out: ODVA Call For Members: Modbus Integration JSIG The kick-off meeting for the Modbus JSIG runs from Thursday, May 10, 11:00 AM to Friday, May 11, 04:00 PM

Side-stepping the marketing fluff and platitudes of a brighter future such meetings evoke, small third-party suppliers and the folks on the plant floor can expect the following benefits. Regardless of the directly stated goals of ODVA, Rockwell, or Schneider-Electric, small vendors will implement solutions that include these abilities:

Vendors making Ethernet Modbus/TCP products will have a simpler "first step" to adding full ODVA/CIP support without the somewhat overwhelming task of 100% conversion of a word-array device model into hundreds (or thousands) of CIP objects.
ControlLogix PLC will be able to connect through Ethernet-to-Serial devices to multi-drops of Modbus/RTU slaves. For example a user with a dozen small Modbus/RTU PID loop controllers will be able to add an Ethernet-to-Serial device to read via Modbus and cyclically produce a small block of word data from each loop controller over Ethernet.
HART, Bluetooth, ZigBee and other new technologies which offer Modbus interfaces will find a instant place as sensors and I/O within CIP and Rockwell systems.
Since Siemens, GE-Fanuc, Omron, Mitsubishi and most major PLC brands offer some method to act as Modbus slaves, users with any of these PLC will be able to integrate them within the CIP Producer/Consumer system.

Looking forward a year or two, I also can foresee some other secondary benefits evolving from this JSIG's work:

I started working with ODVA Ethernet/IP almost 8 years ago and still as-of today the legacy PLC5E, SLC5, and serial MicroLogix (the old PCCC-based PLC) don't have effective inclusion within CIP Producer/Consumer systems. Since the device model of PCCC PLC shares much in common with Modbus PLC, it is a very small enhancement to add a similar support for AB PLC - perhaps AB will actually extend this to future firmware updates to Ethernet-based PLC. In fact, since my Digi One IAP code already allows Modbus masters to query DF1 and CSPv4 slaves as-if Modbus slaves, as soon as Digi adds Ethernet-to-Modbus support per this JSIG's output users of older AB PLC will gain access to CIP Producer/Consumer systems indirectly as honorary Modbus slaves.
Today legacy Modbus and Modbus/TCP systems lack any simple form of multicast producer/consumer exchange. While the IDA protocol offers this, IDA is so many orders of magnitude more complex (and resource hungry) than simple Modbus as to become really an "unrelated" protocol. Any specification that defines a "server interface" naturally implies a corresponding "client interface". So although this ODVA JSIG is not planning to define how Modbus "peers" could use multicast to exchange cyclic data, the end result will be a fairly natural and multi-vendor method to do this. So while I doubt many pure Modbus/TCP products would implement ODVA protocols just to gain this multicast exchange, any products which add the CIP support anyway will naturally add the last few bits of code required to enable pure Modbus-to-Modbus multicast via the ODVA mechanism.
Taking the above point to its natural conclusion means Modbus/TCP masters which implement the Modbus JSIG's "server" function will also gain a mechanism to access CIP Producer/Consumer systems. Even if the ODVA JSIG doesn't cover how to do this, natural methods will be inferred, produced, and copied by vendors to make this a fairly common new product feature.

And lastly, it will interesting to see what response PNO considers to likewise include Modbus slaves more formally into PROFINET. Personally, I have always felt that the simpler PROFINET IO protocol would fit very naturally with multi-drops of Modbus slaves acting like plug in I/O modules ... each slave would be a module within the IO device.

Rockwell and Modbus Data Traffic

2007-04-25T10:09:00.000-07:00

Summary: I compare the data costs for four common off-the-shelf PLC protocols to a remote cellular site. As a rule of thumb, even if you have an Ethernet-based PLC your lowest data costs for SCADA-style periodic polling are obtained using serial protocols via the PLC's serial port. Their modern "Ethernet protocols" are very wasteful of cellular bandwidth.

PLC Protocol Example:
A simple but realistic SCADA scenario is to poll every 15 minutes and read 10 words of data and write 2 words of data. This commonly requires 1 Read command and 1 Write command (I'll ignore the rarely supported Modbus command that reads & writes within a single command.)

While there exists special SCADA protocols and special products that optimize remote traffic, I am not looking at those protocols at the moment. Instead, since cellular and satellite access to remote IP and Ethernet products has enabled people to use off-the-shelf PLC technology, I am looking at the more traditional PLC protocols. These are things which affect users when they apply an Ethernet design to an IP-based wide-area-network system.

I compare these 4 PLC protocols:

AB/DF1 Radio Modem (RM) encapsulated in UDP/IP. DF1 RM is basically DF1 Full-Duplex with no ACK/NAK and is supported by the SLC5 and MicroLogix line.
Modbus/RTU encapsulated in UDP/IP. Modbus/TCP within UDP/IP is roughly the same size.
AB/CSPv4 in TCP/IP as supported by SLC5/05 and PLC5E MSG blocks.
AB/Ethernet/IP as moved by ControlLogix Explicit MSG blocks to PCCC-based remote PLC. Note that Ethernet/IP "I/O Messaging" does NOT work through NAT'd wide-area-networks since the protocol embeds IP information within the data packets and is thus is "broken" by NAT.

The bytes per 15 minutes includes the IP headers, any PLC protocol overhead, and the actual data moved for each update (always 20 + 4 bytes). The MB per month is just defined by 30 days of such polling, or 2880 updates at once per 15 minutes. The 2 serial protocol have been encapsulated into UDP/IP and I assume the remote IP end-point can handle connecting this to the PLC's serial port.

Protocol	Transport	Per 15 Min	MB per month	Relative Cost
Ethernet/IP	TCP/IP	1202 bytes	3.46 MB	100%
AB/CSPv4	TCP/IP	960 bytes	2.76 MB	80%
Modbus/RTU	UDP/IP	166 bytes	0.48 MB	14%
DF1 Radio Modem	UDP/IP	194 bytes	0.56 MB	16%

The two Rockwell "Ethernet" protocols cost a lot more to use in part because they force use of TCP/IP, and therefore suffer the repeated cost of TCP socket opening and closing, plus extra TCP acknowledgment overhead. They also suffer because they both involve connection registration and service functions that needlessly repeat every time the connection is reestablished. While the actual data packets of these protocols are roughly twice the size of the serial encapsulated protocols, the real burden they suffer is all the extra TCP/IP packets exchanged that do NOT directly involve field data update.

Both the serial Modbus/RTU and DF1 Radio Modem benefit that they move no IP packets that don't relate to the field data update - no TCP/IP open or close or acknowledgement; no protocol "service function" overhead. Each moves just 1 read request and 1 read response, plus 1 write request and 1 write response.

Discussion of Other PLC Protocols:

Most other PLC Ethernet protocols will either approach the costs of the AB/CSPv4 - or they won't work at all due to use of direct "Ether-Types" and lack of IP compatibility. Most serial protocols with roughly either match the 2 show here or be twice the cost if protocol ACKs are used by the protocol.

Modbus/ASCII will almost double the cost of Modbus/RTU since each data packet is roughly twice the size. But this wouldn't increase the IP overhead any.

Using DF1 Full-Duplex instead of Radio Modem would effectively double the cost over DF1 RM since DF1 Full-Duplex moves the protocol ACK/NAK, which doubles the IP header overhead also. Using DF1 Half-Duplex would triple or even quadruple the costs since HD not only moves protocol DF1 ACK/NAKs, but the ENQ/EOT polling overhead.

Most other protocols I am aware of - such as Omron Hostlink, GE-Fanuc SNPX, and Siemens PPI - would cost roughly 2 to 3 times more than Modbus/RTU or DF1 RM since they include protocol ACK, while a few even encode many parts of the message as ASCII or BCD form instead of as binary.