Then we got a new client, one that would change the trajectory of our company and my career as a developer. Our client was a growing company that delivered cooking oil to restaurants. Their system consisted of two big tanks behind a restaurant filled with oil (one clean and one used) and a pump that sent the clean oil to the fryer and took out the old oil. No more having kids handle JIBs (jug in a box) full of oil. The oil cost a bit more, but it had the advantage of a) less risk of a teenager burning themselves and b) the oil actually got changed because it was easier.
Back in the mid ’00s, I was working at a small start up. We all met at Andersen Consulting and had worked on big projects together. Our new company did all sorts of stuff for various clients. We had clients in shipping, wholesale apparel and repping manufacturers to Target.
When we first met our clients they were at the stage where they had grown enough that their IT infrastructure wasn’t working anymore. Tons of Access databases in various departments or just plain spreadsheets. We helped them consolidate and grow into better solutions. After we had successfully delivered on these projects they asked us to take on the task of handling all their telemetry data. Each of the new oil tanks had a float sensor that was connected to a controller. The controller would use a modem to dial out once a night with the current amount of oil in the tank. The controller also had a rudimentary ability to raise an alarm if the oil dropped past a certain amount. The alarm was to prevent the store from running out of oil.
Our client wanted us to create a better system for collecting all these tank readings. Their clients were not happy having to keep a dedicated land line available for the controller to dial out on. The client was getting the data from a different vendor in the form of a spreadsheet that they’d manually import into a very simple MS-SQL table. Access to this data was very primitive.
The final solution we came up with was a huge project. We had to refactor every aspect of their old system. The remote devices had to be updated to utilize local networks, the modem bank had to be replaced with a new web based API, new data models had to be created to persist the data and finally new user interfaces had to be created to share the data to multiple user groups.
We found a local device manufacturer that had several devices we could use to phase out the remote modem. The initial phase was adding a small device that connected to the legacy tank controllers via a serial port, then converted the message to an http message that was sent out via the local stores network. New installations all used a new controller that sent the tank sensor data out as a http message natively. Both of the new controllers were also capable of 2-way communication. The 2-way communication was used to manage alerts. A remote manager could push new alert rules to the remote controller.
On the server side, we had to create an application from scratch that collected both the scheduled call ins and any alerts that were generated at a remote site. A lot of time and effort went into figuring out how to handle bursts of messages so that we didn’t miss any messages. The alert handling application also had to interface with various email or SMS systems to dispatch alert messages to the appropriate people.
In the database we had to develop data models that supported not only the tank readings, but information about users and access levels. The alarm rules were also stored in new tables.
The project also created a few basic user interfaces for distributing information about the tanks to various internal user groups. The initial views were pretty basic (but were still head and shoulders better than what the replaced). This part of the project quickly grew in scope. The client was so happy with how much easier it was to distribute this data, they immediately wanted to create more views and make them fancier. We eventually created UI’s not only for internal use, but also for their end customers. End users could log in and see how much oil were in their own tanks and manage their own alarms.
The solution we came up with for our client was really a pretty sophisticated multi-tenant IoT platform. We were able to leverage it to monitor not only oil levels, but walk in cooler temps, drive thru performance and other kitchen equipment. We also were able to build algorithms on the remote devices that could identify drive thru performance or when a fryer’s oil was filtered correctly.
What I learned building that solution was pretty much the basics of any IoT project. I built all sorts of other IoT apps after that, but they all used the same basic components. A remote device sending in data, storing the data as fast as it arrived and then distributing that data via the web.
Fun fact: Our client sold futures on the used cooking oil to biodiesel companies. If the price of gas was too high, the biodiesel companies bought the oil and used it. If the gas price was too low, then it was sold to poultry farmers who mixed the used oil into the feed given to their birds.
I’m going to close with some basic lessons we learned from this first project. There are more lessons, but I will go into those in future articles.
Data is King
Every project is focused on collecting and storing the data collected from the remote sensors. How the data is collected is far less important than what is done with the data afterwards.
Who owns the data is a critical decision. Many clients don’t want to incur the expense of operating databases that hold the data. They simply want to be able to use it. They are happy to have the IoT provider run the database and store that data for them. This saves them a lot of money, but it leads to a situation where the IoT provider now has them by two handfuls of the curlies. Mad at the IoT provider and want a new provider? Good luck getting your data from the party you just fired. And remember you chose to let them hold the data because you didn’t have the ability to self-host. Good luck handling that data dump. Sales Force – while not an IoT company – is the king of this. Start for free, start using it a lot, eventually you realize that they own all your critical data and there is no way you are going to be able to leave.
Data can also be sold several times over. Each project starts with a primary user who wants to utilize the data. If successful, there will soon be a lot of other groups that will want to see that data too. In the example above, our initial user set was the internal managers at our client company. They used it for billing and routing. As we collected the data we were able to sell that same data to the individual restaurants as well. The store portal soon began generating revenue for our client.
IoT was the start of the subscription business model. It is easy to get started with a cheap subscription to your data. Eventually however, you realize that your core business is hostage to your provider. Our client eventually realized that we owned their future and bought us out. It worked on us because we were relatively small, but you can’t buy out Sales Force after they have hoovered up all your data for years.
Security Sucks
All IoT projects have shitty security. There are always challenges to secure the various components of any IoT solution. For a solution to be secure each component needs to be secured. There are a couple reasons:
First off, there is physical stuff that needs to be secured. You are putting some piece of equipment out in the field and it needs to be secure. The equipment is probably built by some other company, so you have to hope that their device has the appropriate security built in. This equipment is going to probably be accessible by rando’s out in the field, so you better hope that there isn’t a reset button or other goofiness.
Next, you have to secure the application being used to capture all that data. This is basic stuff, but it is still something that you need someone focused on. Updates need to be applied, PEN testing needs to be done, audits performed.
The network connections also need to be secured. Does the data need to be encrypted? That costs money and adds complexity. What about firewalls? When we were working in the fast food industry, we often had to deal with the “IT Director” of the restaurants we were planning installs to. We needed to have our local device white listed so it could connect to our servers. You can only imagine dealing with the IT guy for 5 Wendy’s in Kentucky.
Then there is the user interface to the data and remote devices that needs to be managed. You need to have a security model that a) keeps people from viewing data that they have no rights to and b) allows for the self-management of security. There is no way that you can make money managing the access to the oil data for the 5000+ McDonald’s stores internally. You have to be able to give one person the rights to manage access for each store themselves. Tools need to be able to handle this and in a way that is simple enough for some Assistant Manager at Taco Bell to understand.
Field Deployments and Installation
I haven’t really worked on any IoT projects for the last 6 years or so, but I bet that this is still the biggest missing piece in any IoT solution. Mostly because it is hard and unglamorous.
Here is the challenge, you have to put your remote device in a box, ship it to some random place and then hope that the person out there in the sticks can successfully connect it up.
This is one of the reasons that security is so shitty. A properly secured remote gateway/device and sensors are much more difficult to install by a normie. Much easier to simply skimp on security. This is where sensitive information like passwords and IP’s are hardcoded in the gateway firmware. Or local wifi passwords collected during setup are stored in a local text file. It is the only way the developer could figure out how to get something installed remotely.
Even when you have a semi-trained installer (aka Comcast guy in a van), installation can be a challenge. Almost none of the remote devices have any sort of display. So the installer needs to be able to look at some LED’s and know what the status is. Or more sophisticated devices will have an http server running locally that can be used to configure the device. Of course, that means that the installer has to be able to figure out what the local IP is. (Worse, they have to have a crossover cable to connect directly.) On one project, I was helping develop a new remote device and I tried to advocate for adding a NFC chip to it so that local configuration could be streamlined. I was shot down because it would have added a dollar to the BOM (and it was a feature that no one had asked for).
Direct to consumer devices are even worse. Imaging having to come up with a solution where those IoT lightbulbs can be easily found by your mom or dad and configured/installed. Yeah, anything they can handle is going to be open to exploit by any hackers nearby.
Why hasn’t this been solved yet? Mostly because it is a one time pain. After your big disastrous roll out, you forget how bad it was. It is also something that first time IoT implementer forget about. They don’t even think about it. As a veteran of several national roll outs, I would tell anyone who asked that I would buy any IoT gateway that had a good solution for local installation.
Local installations were further complicated by how sensors were connected to the gateway. For example, we were working with a solar company that was connecting the panels on the roof to the gateway via Zigbee (a mesh networking protocol). It worked pretty good until we started installing in California. Zigbee will connect to the first controller it can find. In California, PG&E’s meters were also using Zigbee and the solar panels would often connect to the Zigbee controller on the meter instead of the solar panel controller. The only solution really was to join the solar panels to the gateway back in the installer’s shop before sending them out to the job site.
I enjoyed reading the account of the guy who hacked his smart vacuum so it can work offline.
Same, I actually re-read it a couple of times. That was a solid article.
He should sell the code for it.
That was a good one.
You’re welcome.
You can only imagine dealing with the IT guy for 5 Wendy’s in Kentucky.
You had a line item in the budget for pot and/or ‘shine bribes?
There were two flavors of those guys. The incredibly incompetent who didn’t care and the incredibly incompetent who thought they were the smartest security expert ever.
The apathetic IT guy was the better to deal with. The only hurdle was helping them figure out how to add a new rule to their firewall allowing your device to call out.
The self-proclaimed security guy was the worst. After he initially rejected your request for the firewall update, he would go into a long presentation about all the layers of security he had added to their networks. I would try to gently get him to add the new rule first. Half the time, though, I’d have to get the franchise owner to personally tell him to add the rule or be fired.
JFC, those guys…
I once had to drive from CLE to Rochester (and back) in a day because a customer’s (self proclaimed, it was in his work e-mail signature) “IT GOD” fucked up their entire network with a registry change. I got there, walked in, asked where the machine was, sat down, opened RegEdit (which prompted a response of, “Oh, you’re using RegEdit to make the change…”), changed the hive, confirmed the fix, and was back out the door in 15 minutes.
I was not a happy person that day.
What did he expect you to make a regisstry change with?
Butterflies, of course.
*sprays bug killer*
That’s resource wasteful. There’s no time to have a nerd measuring contest.
“I once had to drive from CLE to Rochester”
I’m holding my sympathy vote until I know which Rochester it was.
My first thought was New York, since it’s within feasible driving distance for the Land of Cleves. It’s also my default Rochester regardless.
I was going to say, I’d rather deal with the single IT guy for five Wendy’s than trying to get a policy change in a F100 with broken processes.
UCS:
No idea, which just made things worse. I had emailed over the exact hive and what to change it to, I have no idea what he did to break it as it was rolled back before they called and complained that “the change I had sent them brought the system down”.
Raven Nation:
It was the one in NY, 4-5 hours away one way.
@Neph: OK, moderate sympathy. More than if it were the one in PA, less than if you had to go to MN.
This is really really fascinating. I haven’t the vaguest understanding of the practical mechanics of making it work.
And-
Sales Force – while not an IoT company – is the king of this. Start for free, start using it a lot, eventually you realize that they own all your critical data and there is no way you are going to be able to leave.
Every time I see a “free trial” offer (I think Quickbooks might have been the first time it really clicked for me) I ask myself “How do you get that info back?”
Yeah QB can suck it.
I’ll NEVER use their cloud.
Jimbo, you seem to have a very interesting job.
The single absolute best part of IoT projects is talking to the experts on the process you are trying to update.
One owner explained to me that the reason that his best drive thru worker had paper clips next to him at the window next to the till was because it was a way to keep track of how much he had skimmed. The owner said that the worker would skim from cash orders. Instead of putting the money into his pocket, where he could be busted with it, he put it in the till, but put a paperclip next to the till for every $5 that had been skimmed. When he counted out the till, he would take out the appropriate amount of money. I asked him why he didn’t fire him if he knew what was going on. The owner said the guy was awesome at working the drive thru and if he wanted to skim some money, that was fine with him. He also said the drive thru guy seemed to like stealing more than getting a raise.
A lot of people hear, “IoT” and immediately think of stupid bullshit like giving your washing machine an internet connection (actually, there are some pretty obvious good reasons to do that.) They don’t think of the commercial/industrial projects that use the same systems, like you described. One that I worked on briefly was a methane reclamation system for landfills – that was pretty interesting.
EvilSheldon – I especially love the cripplewear IoT Bosch dishwasher I own.
In order to to do delayed wash you have to use the phone application. OTH, the top line model has a button to do exact same thing. This way Bosch gets treated to all of my dishwashing habits. When I wash, how much rinse aid I use, what cycles, etc.
Related – I just spent about 6 hours in total last weekend putting new bearings in my 20 year old front loading washing machine. I would not recommend this project, although $80 plus $25 bearing driver beats $1,200 new washing machine that is guaranteed to have a companion phone app.
Sensei:
All that data collected from your washing machine is fed back into the manufacturing process. I’ve seen some pretty interesting things get sifted from all that data being sent in. MTBF data can identify a part that needs to be replaced in new models.
That app is just a teaser to get you to send that data.
When we expanded from just oil tank data to things like walk in coolers, we were able to sell access to that data to both the manufactures and the local service companies. The manufacturers would use the data to improve their products. The local service companies would use it to schedule preventative maintenance or respond to an outage with the correct parts.
What are the obvious reasons on the washing machine?
I can’t think of any.
Pope – Oh I can definitely see the value from the manufacturer perspective for this data.
Part of Tesla’s success is the huge amount of data they get from their autos. They were designed from the ground up to collect this data about everything in the vehicle and send it to the mothership.
I can definitely see the value from the manufacturer perspective for this data.
Yeah, that’s the easy one to sell.
It’s the value for the consumer I don’t see, which is why I bought only dumb appliances when I recently had to replace some appliances.
Sheldon hasn’t been in a laundromat recently. Customers wither sit with their clothes or wander back after a half hour.
The machines are rarely even in fully working order, and added complexity just means more breakdowns.
Their margins are so slim that fancy tech is just going to put them out of business.
And most machines not in use are either A: Broken, or B: Empty.
ES, makes sense on the industrial end.
As a homeowner it’s just odd data mining.
NY is boring ossified and strangled by regulations.
When I lived in TX, there was a chain of laundromat/bars. There was no seating in the laundromat proper, but burgers, beer and pool tables on the bar section. With those pager thingies when your laundry was done.
Last time I used the laundry at an extended stay hotel, it didn’t take quarters. Required a third party app that handled not just notification but payment. Prices weren’t unreasonable and worked fairly well although I think the timer is forecast not real time from the machine. Version 2.0 update.
Re telemetry projects. I work in telecom and have been in it long enough to see the shiny new thing become long past vendor support. Same with legacy data networks. IoT will be ugly in 30 years.
Gustave Lytton:
I was thinking of addressing this in a future article. You are absolutely correct and is a great point.
One of the biggest barriers to our start up was the fact that we were tiny and our clients were worried about what happened if we went away.
UCS, I take some of my bedding and sleeping bags to the laundromat about every two years depending on usage.
The four big machines they have are ‘obsolete’ and only two work. They stay locked during their run so you come back before runtime is over so your sleeping bag isn’t stolen.
Adding complexity is adding points of failure.
Every sensor has a gasket and can leak through internally. I’ve been fighting a trick with wiring and sensor problems all year. Nothing wrong with the engine. It simply decides to occasionally stop talking to itself. Were the sensors mechanical it would run fine, only the cam positioning sensor is needed for the computer to function properly, some of the other sensors are nice to have for efficiency purposes but ultimately unnecessary, but they decided that if one of the sensors that are for monitoring go tits up it shuts down. Nothing like hitting an 8 mile long 7% grade and you can’t downshift because the throttle went dead.
Why is the throttle electronic instead of a cable? Reasons.
Throttle bodies are electric so the manufacturer can give you 25% throttle at 10% pedal the make the engine feel stronger than it really is. Secondarily, cruise control is cheaper o to implement because the computer can directly control the throttle instead of involving a second cable and vacuum actuator. Thirdly, it’s a lot easier in tightly packed engine compartments to run wires than cables, which require gentle bends to avoid binding. Fourthly, throttle by wire is more reliable since it uses a dual sweep sensor rather than a cable that can lose a swage or get fouled with debris. Fifthly, it’s more expensive to fix for the consumer which makes the dealer network happy, and when enough expensive parts break you’ll buy a new car instead of operating it with bread ties and prayers.
My 350Z has an electronic throttle. I has performed flawlessly for 19 years.
I was thinking on a big truck where room is not an issue.
Chevy seemed to have throttle cables figured out starting in 1971, the only poorly ran cable I ever experienced was on my 93 Dodge with the Cummins. Not only was the structure of the pedal too week for the job but the angle at the firewall was wrong, the hole needed to be 3/4” higher than it was and dodge managed to think that having a loop in the cable was a good idea.
Like the electronic parking brake controls on so many modern cars, there are some things that need a physical linkage as a margin of safety.
Kennith, the E throttle wasent the problem, an engine sensor failure telling the computer to cut off the throttle pedal was. If the throttle pedal were mechanical the sensor failing wouldn’t have had any impact on throttle control, a safety issue.
I don’t know if the early cable controlled LS engines had a vacuum controlled cruise or not. Those manifolds are popular for swap guys though as they are much more user friendly.
Place I worked at many years back had a fleet of delivery trucks that had in truck computers (some bolted in, some removable with a cradle) that they would get their routes and dispatch messages on. These would communicate with a little Cisco WAP (Wireless Access Point) that was in “the shed” (or garage, usually a shed was more accurate). The drivers weren’t big fans, because at the end and beginning of their shift, they would have to make sure that they were close enough to upload/download their shifts.
This meant lots of WAPs, networking equipment, and terminals out in the field (this was all backended by an ancient VAX system). We used Cisco’s (fairly nice for the time) management system to set up the WAPs, so they would come in, we’d plug them in, and ship them out. One of the managers decided that he was going to make things better, and turned on remote management for the ones in the field that were already installed, so that if there was a global change, or even regional ones, he could make them from the home office.
This worked great… up until someone fat fingered a configuration of a new device, and pushed out a bad build to all of the WAPs in the field (setting them to a static IP address that wouldn’t be reachable). All of us in the IT department were tasked with getting to every location in our area (Ohio and Michigan) and reconfiguring the WAPs.
Went through all of that at Iron Mountain when we went to a “paperless” system they purchased from UPS. But, they cheapened out, all of the WAPs had issues, we needed to go back to paper 2-3 times a week to meet our service deadlines, the company who was handling the tech rollout couldn’t meet their deliverables, and so on. The whole thing was a disaster.
It worked marginally better when I went over to RRD, mostly as we contracted a tech company, who in turn contracted all of the drivers, making sure that they were on the hook for any issues with the system. But, they couldn’t handle a lot of the driving issues, didn’t know good from bad when it came to the people they hired as management for that end of things, and the whole thing swirled the drain until RRD bought them outright.
Division of labor, how does it work?
Fun read, thanks!
My issue in financial services is data quality. Both external and internal. For example you can have client data across products with a differently formatted address or name as well as from third parties we interact with.
It’s the reason that people go all in on “clean sheet” startup tech heavy net focused financial service companies. However, those companies generally stink at actually creating product.
One of the things that makes IoT a good thing is that it can improve data if done properly.
For example, if you add a sensor to a walk in cooler in a restaurant, you will get a much better idea if the food is being properly stored than you wold with the old clipboard on the outside of the cooler that is filled in by a worker looking at the thermometer mounted on the outside.
When I was helping pitch these projects, I could do an entire presentation on how bad the old method was (of course, the manager would fill in 8 entries at a time instead of doing it once per hour like they were supposed to). The thermometer that came with the cooler was a) located right next to the door which is the warmest part of the cooler and b) about as accurate as the thermometer that comes with your home grill.
We had a whole booth demo showing that by adding multiple sensors (air and solid) throughout the cooler, you could get a much better idea of how the cooler is performing and – hopefully – immediately react if the cooler broke. Each cooler has about $15K of product in it that has to be thrown away if a certain temp is exceeded. With constant monitoring and alarms, you could move the food if things went bad.
Anecdote:
We had one walk in cooler that would have temps climb every night for about six hours. Then it would go back to normal. The local manager investigated and found that the night dishwasher would get hot and open the walk in cooler door and let the cold air gush out while he scrubbed. A minor gold nugget that IoT discovered.
Thats what I did for my building automation company. Coolers lights and hvac, fun stuff to build and commission.
Bobbo,
Building automation is one of the areas that IoT is a good fit for. There are a lot of good things that can be done to increase efficiency and save money.
The projects that are successful are when a building management company handles the connectivity piece (they install IoT gateways and access to the internet), then open that to various subcontractors.
For example, you would connect all your HVAC stuff to the common network and export the data to the building management’s data store.
The projects that fail or are semi-successful, are the ones where a big IoT company comes in and promises to automate everything. All these IoT vendors grew from a smaller company that did something successful in one area and while doing that came up with a platform that can sort of be extended to other areas. An HVAC company that now thinks it can easily collect other stuff like motion sensors in bathrooms (to manage lights). These rarely work because the system was designed for one specific thing, and now is being tortured to do other things as well.
As Oracle and SAP have proven over and over again, there is a lot of money to be made from companies that want to implement a supposedly one size fits all solution (only to discover that to really work, you need a lot of billable hours from consultants to fix things).
I only had time to glance over this, but it sounds a bit like the things I did early in my career. I worked for an industrial automation company, and we worked with steel mills and manufacturing. We had guys that worked with the sensors and controllers, my job was to interface with those things, store the information in a database, and put them on a pretty screen. That’s where I learned that the processes most people see as dirty or boring can include incredible feats of technology.
TOK.
Yup. Creating dashboards was a big part of what I did.
A lot of people who made IoT equipment were pretty clueless about how to sell it. They could hook everything up and collect all sorts of data in the cloud, but weren’t able to present that data to anyone in a compelling way.
I learned just enough about the hardware to get things installed, but spent most of my time talking to end users about what was important in the data, then try to create a web page that displayed the stuff in the format that they wanted.
A typical engagement would start with some customer telling me that they had an Access database that had been created to pull data from a central database. Then someone in Dept A would open the Access db, open a table and copy/paste the data into an Excel spread sheet. Then that spreadsheet would be emailed to others in Dept A. Those other people would all run various formulas to create graphs and other stuff. Those new spreadsheets would then be emailed to others.
I’d have to figure that process out step by step. Then I’d show them how instead of all those steps, I could have people go to a web page where all those reports would be available. I would also allow them to filter the data by dates, or other metrics. People would love this.
The next step would be them coming up with other reports that they could view online.
I’ve spent quite a bit of time supporting Cisco phone systems. The Cisco phone system is an amalgamation of acquisitions, hacked together bits, processes that linger because no one wants to change them, and inertia.
Quite a few of those systems interact through what’s called a PG (Peripheral Gateway). Those were the real workhorses that converted data and allowed the systems to work together.
The final solution
phrasing …
We were able to leverage it to monitor not only oil levels, but walk in cooler temps, drive thru performance and other kitchen equipment.
this sounds like marketing 🙂
Pie,
When we started that project, there were almost no public wifi access points. If you left your house, you were pretty much without network access.
We went with our client to a big meeting with McD’s and afterwards we pitched the CEO of McD’s to leverage their stores into a subscription network. They already had wired all their stores with broadband access (a big deal back then). For a smaller cost, they could have added a network access point to each store. Then sold a subscription to that network. McNetwork.
CEO looked at us like we had three heads. Oh well. I still think that was one of the ideas that could have made me millions if I could have sold it.
Which one of you owns this place?
https://www.realtor.com/realestateandhomes-detail/2249-Lonsdale-St_Camarillo_CA_93010_M28838-72119
For those looking to buy. https://m.youtube.com/shorts/e4uydxO686M
They seem fun.
‘Fun!’
Has potential but it is in California.
Which Glib is this?
https://x.com/iowahawkblog/status/1986444940532977941
What kind of sick fuck doesn’t put down a rag underneath the guns to prevent CLP stains on their cot?
*raises hand*
I was one of those teenagers who burned himself (slightly) while changing the deep fryer oil. I stripped down to my tighty-whities right in the kitchen. The cute waitress was unimpressed.
Can you blame her?
There are some real nasty stories about someone trying to replace the oil in a fryer and ending up getting burned really, really badly.
Not to paint owners of fast food joints with too broad of a brush, but I think most of them were more excited about the idea of the oil actually being changed regularly than they were about worker safety.
When I worked at the local Perkins as dishwasher as a teenager, one of the nightly tasks was to clean the grease trap. No dishwasher ever did that voluntarily ever. We’d all initial the sheet that said we did, but that was a lie.
The grease trap would back up at some point and there would be a minor flood in the back. Whoever was washing dishes at the time had to clean the mess up. The entire time you got stuck cleaning up, you would be muttering curses at all the other lying dishwashers who clearly didn’t do their job.
Thanks for the read, Holy Father. I’m actually going through the course for Ignition SCADA right now in hopes of a career change, so this hits home for me.
Thanks. I hope it works out.
There are going to be a lot of jobs for people who are savvy in IoT installation/maintenance. Like Bobbo above, HVAC workers have to be pretty good working with controllers and sensors now.
One other aspect of IoT stuff in restaurants that I should mention: The Black Box with a Blinking Red Light Effect.
I used to pontificate about the fact that once the workers realized that all the IoT sensors/apps worked, you could replace it all with an empty black box that just had a blinking red light on it. You only needed to tell the employees that you were now able to monitor something and they’d believe you.
We did a filter monitoring project once. We had sensors on various valves and switches and came up with an algorithm that decided if a fryer had filtered its oil properly. We got everything working in a lab and then deployed it to a pilot site. Our customer picked the pilot store. According to him, they were his best restaurant. They did everything by the book.
The roll out was rocky. The techs who installed the sensors and IoT gateways, ran a filter test while on site and everything was great. Over the next week, we barely got any data from the fryer. When we did it never matched a “good filter”. We started sending out techs to look at the install. The techs were always able to run a test successfully. When questioned the restaurant staff swore up and down that they had been filtering once per shift. The owner backed his staff to the hilt.
Finally we sent out some secret shoppers who watched the staff and verified that the fuckers weren’t filtering ever. A few times at the end of the day, they would circulate the oil (but wouldn’t do the other steps for a good filter).
We had a come to jesus meeting with the owner and staff. The owner was embarrassed, but switched to our side and read the staff the riot act. After that meeting we started getting data just like we expected. The owner told us that the staff started doing all sorts of other stuff better as well. They were all convinced that they were being monitored at all times, so they’d follow all the other rules as well as the filtering stuff.
My joke was that you could just put a magnet on the trash cans in the bathrooms and tell the staff that they were now being monitored remotely as well. And it better never get more than 90% full. After the staff had been caught by some other IoT stuff, they’d totally believe it.
Never worked directly on any kind of IoT projects. I am a veteran, however, of several dozen failed bid & proposal efforts to bring some level of IT to air transport aircraft to help optimize airline operations.
This is not related to flying the aircraft. Instead we dealt with more mundane problems. Like you must have enough toilet paper in the lavatories to dispatch the aircraft. And how does anyone in the back office know how much toilet paper is on all 1000 aircraft in the fleet? Well, someone writes it down on a piece of paper that gets handed off multiple times until it get entered into a computer and then someone determines that the aircraft needs more toilet paper. Or someone goes out to the aircraft with a cell phone or walkie talkie and then calls back if the aircraft needs toilet paper.
There are certain complications in bringing any kind COTS IoT device to an aircraft. You need to be able to make rational arguments to the regulatory authorities that your device can’t trigger some sequence of events that results in 250 people getting killed. And by time you get done with that process, your IoT device is obsolete. The supply chain will dry up, and you need to start with the next generation of devices.
And anyone that touches or uses your IoT device and/or its data is a member of a union. If your devices makes life harder for someone, it will go nowhere. If it can replace union members, it will go nowhere.
Oh, and we haven’t even begun to talk about who owns the data being generated at the aircraft.
Note that all my experiences with this are 20+ years old. I am certain that someone has figured this out by now. Which is why the airlines are rolling in money.
Kinnath:
Uffda. Yes, there are places that IoT can work but is always thwarted by regulations. I’ve worked on projects that failed for exactly the reasons you outlined above.
We had a client who wanted to get into home health care. They thought that there was a lot of money to be made by instrumenting various things that could prove that the patient had received the care that they should.
Our clients were a couple of crazy entrepreneurs who had made tons of money in other fields. They – and us – were humbled by all the regulations and red tape that go into home health care. We got a couple pilot sites set up and were able to verify when care provider showed up and left. As well as monitoring a few things to verify that they were getting the goods they were supposed to. Things actually worked.
Things went off the rails when we started talking about rolling it out. The health care workers revolted (the pilot project had had volunteers), then it turned out that the automated reporting wouldn’t be accepted by the FedGov because they hadn’t certified any of our data. The cost of certifying things turned out to be cost prohibitive (if not impossible).
Yeah. Nuclear energy and health care. Those folks live with huge regulatory burdens too.
I had my mom on home health for the first few months of this year. Once she and I decided she wasn’t actually dying, I had to ask hospice how to proceed. They said, “Well, you’ll have to find a PCP.” The fuck. She’s bedbound. No way she’s going anywhere. So I googled docs who make house calls.
This one guy in the metro has an entire infrastructure of providers and COMPANIES of providers that revolve around him. They have a really good system. I don’t know how it’s run, but I was very impressed.
Now, one thing they COULDN’T do was spend time getting my mom mobile, so I hired a mobile physical therapy provider. They didn’t get her quite mobile, but they did pull her a long way forward.
We were developing a WiFi access point to install on the aircraft. Transmitting RF inside an airplane is a really big deal. We had settled on the latest standard for frequency hopping spread spectrum for performance and robustness in a long narrow tube (can you say multipath).
By time the certification testing was complete, the industry said “yeah, never mind — direct sequence is the answer”.
Shoot, airports are just as regulated.
I was working on a project to build an airline reservation system back in the mid ’90s. My task was to figure out how to connect a card reader at a gate with our central server. The idea was that you would swiper your driver’s license or credit card and get onto the plane. No more paper tickets!
The card reader was pretty easy (Verifone), but transmitting the data wirelessly to a gateway and then getting permission to connect the gateway to the airport network was an impossible task because of all the red tape. And this was pre 9/11 so things were pretty lax.
Another example of a solution that failed because of red tape.
We had a client that made the machine that tested for flu viruses. They wanted to automate things and add a cellular module to their machines so the results could be transmitted in real time.
The process was:
1) Machine tested a sample and recorded results to internal memory
2) At the end of the day, a tech would hit a button and a local printer would dump the results
3) The tech would fax results to the State Health Dept.
4) Someone at the State Health Dept would enter the results by hand to their database
5) Every week, someone at the State Health Dept would dump results to a spreadsheet. Then email the spreadsheet to the CDC
6) Someone at the CDC would add the results to their db.
From the time of testing to the CDC getting the data was 4-6 weeks. Our client told us that all flu testing data was only useful for historical purposes. It was too stale to do anything proactive with it.
The thinking was that if each test could immediately be transmitted to the State Dept of Health and the CDC as soon as it was completed, things could be sped up by orders of magnitude and maybe could be used to react immediately.
The CDC said not to even bother them until we had cleared it with the State Dept of Health we were doing business with. The State grudgingly did a small project to create an API for us to send data to. That data was not put into their db. It was in a separate db and it would occasionally be downloaded and used to start at Step 4 again. The only thing that changed was that instead of a fax, the data entry clerk entered the data from the spreadsheet. So almost no gain in efficiency.
The cost of adding the new hardware to the testing machine also added a bit of cost to it. The govt drones who bought equipment choked on the cost. They definitely didn’t want it.
So even though, this should have been an easy win, it never went past the pilot.
When the Covid hit, I kept thinking of this project. I knew that there was no way that the same bureaucrats that couldn’t be bothered to collect flu data in a timely manner were going to do a good job with Covid data.
This makes me very uneasy.
I try to be careful not to put anything in any database that I’d be sad to lose or can come back on me in some way (except my meme folder on Dropbox, which has questionable -ist content), but you know, everything I use is databases. WordPress [my webhost], Carbonite, Dropbox. Do they “own” my data? I don’t know. That’s why I’m asking. Yes, I know Taco Bell and the other phone apps “own” my data. Does it matter?
I want so very much to self-host everything: email, web, and cloud servers. But it’s just too much and I’m overwhelmed with shit I use and is gradually getting worse with each iteration. The enshittification continues apace. (Yeah, okay, I’ve bawled my eyes out in frustration over the last three days because of SOFTWARE, for fuck sake.)
Also, I’m lazy. I have Tor. I could get a VPN and use it. I don’t.
As for subscriptions, I’m doing everything I can to preserve my physical media, keep my “outdated” software (I’m still using Photoshop 7 and Illustrator 6), and find ways to get around subscriptions. Adobe is the worst. Microsoft is starting to close in on Adobe for that spot.
Every update/upgrade chips away at the user’s autonomy and freedom, and adds more bloat. “Legit” software is bordering on malware, installing something that opens a back door to install something else that opens another back door until it’s taken over your machine. WordPress has completely lost the plot.
I’ve been saying I’ll build a Linux box since 2000. Haven’t done it yet.
In any case, I’m going back to pen and paper when/where I can. I’ve used more manila file folders the past two months than I have in the past two years.
I do NOT want to go offline and I don’t want to give up my tools, but damn. I can’t take much more of the enshittification because it’s all making my life so much HARDER than it used to be, so much harder than it HAS to be. Shit, now I’m crying again.
I feel your pain Mo. I have had to spend a ton of time messing around with a new Windows 11 PC and an Android phone reset recently. I have been incredibly frustrated by the fact that it seems like I have no control over my hardware. Turning off various “features” has enraged me beyond belief.
I should be able to uninstall any app from my pc or phone that I want. I don’t care what you think I need. I don’t want Microsoft Edge for any reason.
I completely understand why subscriptions are so wonderful for the companies selling them. I hate them as a consumer. I want to buy stuff I like and then be able to do what I want with it.
Why I can’t buy a dumb TV anymore.
* Two of the smartest guys I ever developed with have gone pretty much full Kazinski in the last 5 years or so. Flip phones, linux, and other dumb stuff. They are more dedicated to the lifestyle than I am, but it isn’t a good sign when smart people are dropping out.
Two of the smartest guys I ever developed with have gone pretty much full Kazinski in the last 5 years or so. Flip phones, linux, and other dumb stuff. They are more dedicated to the lifestyle than I am, but it isn’t a good sign when smart people are dropping out.
I haven’t gone back to flip phones but sometimes I think about it.
If I could make a living with Linux and flip phones, I would.
I don’t have Apple (except for my iPad) for a REASON, which is to control my own shit, but Android is drifting that way and now I don’t have the option to refuse updates. I can put them off for a while, but eventually, it forces it.
One of the problems is LABELS. They name stuff weird shit that’s not intuitive and it FORCES you to accept them. You can’t find a way to turn X “feature” off because you don’t know what it’s called.
They took away phone jacks. Why? Earbuds. Except, earbuds fall out so then people do this. “Earbud straps.” Like, the fuck. That’s just a phone jack with nine extra steps and fourteen times more money.
They took away micro SD card slots. Why? So they can have easier access to your files.
Man, I just wanna listen to music and read books.
Microsoft’s bound and determined to get me inside their walled garden. Thing is, if I cave, my other stuff will break and I DON’T WANT THAT. Leave me the fuck alone with my 20-year-old software that still works.
The only thing that gives me any hope when I’m force to “upgrade” (read: downgrade) is that by the time I do, someone (or lots of someones) have built a utility to deal with whatever issue I’m having.
It’s a small, stupid thing, but I like having my taskbar on the right. Win11 won’t let you do that natively . Guess what. Someone made a utility for it.
My file explorer’s search function went wonky, which is unfortunate because my file tree can get 10 folders deep. I NEED to be able to search. So I went and found a better file manager. This frustrates me sometimes because this one’s search is so complicated, but it does other things I really like. It has functions that actually replace three other utilities I use.
That said, I like one-purpose utilities. Do one thing. Do it well.
I graduated college in 1995, and from that point forward I had at least a few coworkers who built their own Linux boxes. It is kind of a right of passage. I never did, until about 2 years ago when someone gave me an old Windows laptop and I couldn’t log in to it. So I decided to try Linux. It was so easy. With Linux Mint, you download the O/S to a thumb drive, and you can boot off of that to try it out. If you like it, there’s an installation option when you boot.
While it has its issues, for the most part it boots fairly quickly, and I can find applications to do almost everything at least adequately. I even use a remote desktop app to connect to a Windows PC at my old job when I do side work. Best part is, there are no forced updates and no background processes using up all the memory.
I love Linux Mint too.
If my parents were still alive, I would have moved them to Mint. It does pretty much everything basic users need and isn’t a complete pain in the ass.
The problem with Linux is still support. When there is an issue, you end up somewhere where the Linux fan bois are having some inane technical flame war. Sigh. All I want is a fix. I don’t want to have to sift through a rehash of some argument about the best distro to use.
At this point, the only advantage Windows has is ease of support and proprietary apps (such as gaming). Linux has been making quite a few advancements on that front, particularly with the boost that the Steamdeck has given *nix gaming support.
I agree, especially if most of what you do is in a browser. For the gym we use Canva to make our marketing materials (free in the cloud) and Google Docs for spreadsheets and stuff (also free in the cloud). I have Libre as a replacement for Office.
I haven’t really needed support, other than having an occasional issue with booting due to disk errors, but that’s just a command or two, easily found. As I recall, connecting a printer, network, and peripherals were simple.
I’m perfectly capable of using Linux it’s just too rough. The only device I’m currently running it on it is two Pi boxes.
I’ve yet to run a single distribution where something didn’t break and require an hour of Google and the terminal. And as Jimbo points out the whole community is more interested in flames and not helping people.
Oh, look. I’ve spent upwards of 20 hours this week googling to fix things that shouldn’t have broken across five different pieces of software. This is getting more and more frequent, more and more maddening. Contacting tech support that’s mostly non-existent, bot-driven, or requires a complicated phone tree and deciphering a heavy Indian accent coming from a guy named “Steve,” who treats you like you’re a moron, but the native English speakers also treat you like a moron.
“Why is my virus scanner suddenly stripping the attachments of the only two things I need, when it hasn’t had to strip ANYTHING in years?”
“Because it told you why.”
“I read the reason. It doesn’t meet the criteria.”
“That’s the way it is. If you don’t like it, turn the virus scanner off.”
“Why would I do that??? I want THIS DOMAIN not to have its attachments stripped.”
“No way to do that.”
WTF
“So, it’s an all-or-nothing?”
“Yes. Ticket closed.”
Mo,
I think tech support is way worse if you have some technical knowledge. The utterly clueless have hope as they clear their cookies or turn it off (wait at least 5 minutes) and then turn it back on. If you know something about things, that shit just pisses you off.
I just bought a new computer because my old Dell laptop will no longer cleanly boot. Some update added a check of the Trusted Protection Module (TPM) to the POST. So I have to constantly press C to continue to get back to working. My laptop came out before this TPM nonsense was a gleam in Bill Gates eye, so it is no surprise that one wasn’t found.
I’ve tried to talk to Dell support, but I get some guy named “Jake” in New Delhi that is obviously reading from a script. It becomes clear that the only thing more tenuous that Jake’s grasp of English is his technical knowledge.
I finally got tired of arguing with him about the impossibility of “clearing my cookies” being a successful fix for my problem. I just want him to tell me what version of the BIOS do I need to restore.
where something didn’t break and require an hour of Google
Y’all have a very different experience than I do. (grasps onion on belt). Why I’ve been running Linux for …. mumble mumble … nigh on 25 years and solaris for 10ish before. Even prior to 15+ years ago where it was much more intensive hands on work, never had that many issues. In the last probably 25 years, I’ve been running versions of Redhat (and respins like ScientificLinux, Rocky) then Fedora. This is on laptops, desktop, and yes, servers running Fedora. By far the worst case is laptops as far as hardware support – you still can’t really buy the latest laptop because some chip will have a slightly different version and hence driver requirement and you can’t tell from the re-seller (or manufacturer half the time).
But certainly in the last 10 years, if you buy a one-year old laptop, Linux will install and run out of the box. And for desktops and servers, I’ve just never encountered those types of problems. I don’t think I’ve had a single machine running continuously for 20 years, but that’s due entirely to wanting to upgrade the hardware. Generally speaking, machines have just been seamlessly upgraded without the need for clean installs throughout their hardware life. And almost all the time without having to invest much time at all in ‘stuff that was broken because of the OS’. Shrug, maybe just lucky.
But certainly in the last 10 years, if you buy a one-year old laptop, Linux will install and run out of the box.
It runs fine fresh install. They push any update of reasonable scope and something will break.
You’ll either need to revert some library or run something that’s not on the main branch to get it fixed. Meanwhile there will be huge fight over which forked version of the the now borked library is now the one that everyone should be using.
I may as well have MSFT cram Copilot, OneDrive and Office 365 down my throat. If half the world didn’t work with it I’d just by a Mac Mini and be done.
Putrid Meat:
Back in the day, my issues were usually around wireless cards and getting them to work. That was a long time ago. As you said, most cards now have linux drivers and it works pretty well.
My most recent issues have been with multiple monitors and a linux laptop. I run into minor issues with scaling on the external monitors, or having them randomly re-arrange themselves. I’ll google for fixes. I always hope that there is some simple conf file that can be updated.
I like Linux a lot. Linux fan bois are a lot like bicycle fan bois, they tend to be so nutso that I hate that we both like the same things. When I bought my first macbook pro, I had to do it online because every time I tried to buy one at the Apple store, the “genius” helping me was so insufferable that I gave up.
They push any update of reasonable scope and something will break.
I just have honestly never had this problem. For laptops, there’s usually an upfront cost of doing due diligence on hardware (and not buying the bleeding edge laptop mitigates that almost entirely) but once I have a clean install, things have just worked through the EOL of the hardware (an EOL not enforced by the OS vendor BTW).
multiple monitors and a linux laptop.
As mentioned above, the only real issues I’ve run into are laptops and that’s mostly sorted with due diligence up front, but I’ll concede a potential rocky road with them. With laptop displays, the introduction on the hardware side of multiple graphics cards 5-10 years ago really threw a wrench into the laptop display world (that had been getting much better) for linux. But that was sorted relatively quickly and is mostly not a problem now.
Apple genius
That’s not linux, that’s BSD with an expensive pastel shell over the top. I don’t think I’m a linux fanboi, but y’all might put me in that camp. People who want windows or mac, have at it, it serves your purpose/use case. But I will argue with you. Because it’s fun. Especially Mac fanbois.
PM:
I have had several macbook pros and like them. I get a decent OS and everything works. I get the downside of it too. Want to do something crazy like store your photos on an external drive? Be prepared to fight.
Apple also has done nothing to help their customers since Jobs died. I gave up on buying any new Apple macbooks since they started soldering in the memory and hard drives so you can’t do your own upgrades.
For the most part, I like using Linux on my laptop. I almost always dual boot it. Every once in a while I run into something that causes me to switch back to Windoze for a while.
Provider owns the data and gets to set the security, too. Oops, data breach? Sorry. Use your piece of contract paper to rectify it.
Through the glass display case. Wow.
https://x.com/SteveInmanUIC/status/1985946595968643319
Looks like the shooter got him in the brain.
That is some fancy shooting to hit something that small.
Shutdown the Musical
I like the song at 40:43.
https://x.com/printingguns/status/1986193506713358543
Lol
Nice.
Why is the throttle electronic instead of a cable? Reasons.
So there can be different “performance” settings which vary the relationship between pedal position and throttle opening.
It also makes traction and stability control much easier and better. Trying to use brakes to stop a wheel that is loosing traction is much easier when 400hp is not being applied.
Pope, great article on IoT. I learned quite a bit about the topic of interest, but this line was the most entertaining to me.
“Fun fact: Our client sold futures on the used cooking oil to biodiesel companies. If the price of gas was too high, the biodiesel companies bought the oil and used it. If the gas price was too low, then it was sold to poultry farmers who mixed the used oil into the feed given to their birds.”
Feeding chickens with the medium used to cook and bits of their ancestors. That is dark.
Yet, we also eat pigs, which will gladly eat a pig, so…
Is it called short pork when they ear each other.
Thanks.
I learned that nugget (shpip pun intended) was because our clients were having issues with people stealing the used cooking oil. The price of gas had spiked and the biodiesel car guys were actually stealing a lot of the used cooking oil.
When they were discussing the issue, I asked why it was an issue. I would have thought they were doing half your work for you. That is when they clued me in on the uses of old cooking oil.
We sold the old oil tanks to a company that made pontoon docks from them. I can’t remember the name of them though.
The Simpsons did it.
And also.
Fucking TurboTax won’t run on Win10 machines as of January 1, 2025.
Husband managed to get Win11 on an old desktop machine that was outdated 10 years ago.
Oh, and also, he did build me a Hackintosh also for the sole purpose of uploading ebooks to Apple bookstore. I don’t know if it still works. A client gave me an old MacBook to do it, so that’s what I use IF I use it at all.
When my life begins to wrap up and all I have left is posting here, I will go back to my ancient Linux laptop and use that. My retirement jobs will be outside of the realm of technology.
I bought an HP miniPC that I’m going to run headless for TurboTax. I managed to get Win11 with a local account running (fuck you Microsoft, I do not want or need an online account to log into my local machine.) It is currently off with it’s wifi board in a plastic bottle and no ethernet cable plugged in.
I have to go through and scrub out all the annoying details before I let it near a network.
Early horoscope:
Take next week off, stay home.
I only took monday off.
What’s going to happen wed-Fri?
I have to go in to office Monday and Tuesday for big all-hands meetings. Most likely I will be let go.
Next week:
Monday: USMC’s 250th B-day
Tues: Trying to survive hangover from b-day party.
Celebrate by overthrowing an Islamic country – Minneapolis is close enough to you.
I think you’re 3 weeks late with that one. 😉
Someone explain this to me.
Cafeteria puts out their pie price list.
Apple, pumpkin – $20. Yikes, but all food has gotten expensive.
Pecan – $24. Ok, pecans can be expensive.
Chocolate cream – $24. Wut. Since when are pudding typ pies not the cheapest option.
Coconut cream – $40. The absolute fuck? Someone explain this. Are coconuts coming from tariffed growers or something? I thought they were Hawaiian.
https://worldostats.com/country-stats/coconut-production-by-country/
Hawaii doesn’t appear on the Top 86, Puerto Rico makes less than 200 metric tons per annum.
If you trust the USDA, PR, Cuba, Central America, Pacific.
FL, SC and all those other states with palm trees need to get with the program!
Coconut cream – $40
Do you know how much you have to pay the guy who scrapes the cream from between Kamala’s thighs? I’d say you are getting a bargain. Probably because it is month old cream. Or it isn’t pure. Cut with Gentleman Doug’s cream too.
… what is wrong with you, man?
We can’t count on SF being around forever. I’m trying to develop some skillz to help cope when he is finally locked up.
I dont think Hawaii grows much of anything anymore. Coconut production: Indonesia, Philippines, India. So yeah, tariffs most likely. That should end quickly. Give it a year or so.
If I had to guess, Agriculture has been regulated into oblivion in Hawaii. It’s not theoretically banned, but it’s impossible to make any money from it.
I think that coffee is the big cash crop now.
Well, when you can sell burned bean juice for $2/oz and each tree makes how much?
If it isn’t cheesecake, I don’t want any.
Someone has to train the monkey to climb the tree and retrieve the coconut. Then there’s Humane Society observers who need to be paid. Monkey union reps. Environmental impact statements. 20% coconut set asides for underprivileged monkeys. Monkey health insurance premiums. Hourly banana breaks. You’re lucky your pie is only $40.
Coconut cream – $40.
Jacking off a coconut is skilled labor.
Jacking off in a coconut is a labor of love
I think that coffee is the big cash crop now.
I thought it was Maui Wowee,