How does/did file/record locking work?

[alank2]

I saw there were some functions in MP/M to support it.

And then share.exe in DOS 3+ seemed to be where they added support for it to DOS. Why the TSR? Why not just build it into DOS?

I am presuming early Netware had some sort of file locking - what DOS did that require then? 3+?

Generally speaking how would to programs operate on the same file at the same time? Do they first lock a region or regions that they need to read or write, and once locked then they can be sure that those regions will not change - so they could then be read and know that the other program can't change them?

 

[krebizfan]

Share.exe was a large file which can be seen with the problems of PC-DOS 4. If only one program was running, the need for SHARE was considerably reduced.

DOS 3+ would lock a region of a file if requested and if SHARE or other network was also loaded. See documentation for INT 21h Function 5Ch. Still DOS and still ways to work around the lock on a local drive with generally bad results.

IIUC, MP/M could lock specific records or an entire file depending on how calls were done. Normal for MP/M was that the file would be exclusive but a program could allow other programs to have either read only or complete read/write access. An extended lock would prevent access even after a program terminated.

 

[Svenska]

File Locking requires intercepting file I/O operations and keeping global state for every locked region. This costs memory and some performance, and is generally unnecessary for local, single-threaded applications.

Locking is required for networked file systems (because different clients may access the same file) and even useful for multi-threaded applications (so that threads can coordinate resource access with each other) even in a single-task environment such as DOS or Windows 3.x. This is the reason that Winword 6 or Excel 5 require SHARE.EXE to be loaded in Windows 3.x, even on a local system.

There are different types of locks as well. An exclusive lock prevents anyone else from accessing a file or region at all, but other types of locks can prevent others or anyone from writing while such a lock is active. For example, multiple threads may write to the same log file, so one thread should be able to block anyone else from writing at the same time. But when a thread is updating a database, it causes the structure to be temporarily corrupted, so it needs to be able to block anyone else from even reading at the same time.

Some locks are opportunistic, i.e. they require everyone to actively participate in the locking process for locks to work at all (this is the case in UNIX), while systems such as CP/NET or SHARE.EXE enforce file locking even for applications not aware of file locking at all.

 

[whartung]

Generally speaking how would to programs operate on the same file at the same time? Do they first lock a region or regions that they need to read or write, and once locked then they can be sure that those regions will not change - so they could then be read and know that the other program can't change them?

The most fundamental use case was for shared data files over a network.

The simplest use case is that you had two programs (likely on two different machines) that wanted to add a record to a file. Each program would try to exclusively lock the file. They would try this at the same time, which presents a race condition -- the programs are racing to see who gets the lock first.

Whichever gets the lock first, now opens the files, goes to the end, adds the data, closes the file, and releases the lock. During that time, the other program would just hang. When the first program finished by releasing the lock, the other program would continue as if nothing had happened, and then proceed to do what it wanted to do.

There's obviously a lot of nuance to this. Things can get complicated quickly. A file lock is rather coarse, later programs would just lock sections of the file rather than the entire thing.

For example, with a data file with a fixed records size, if you have program A writing to record 100, and program B writing to record 200, A would just lock that section of the file that covers record 100, while B does the same with record 200. This allows both programs to write to the file simultaneously, without having to wait for each other, or stomping on each other.

Lock management is all sorts of fun when you have programs lock things and then crash, without updating the lock manager. Or the old days when Sally would go into the Customer Master, view Consolidated Industries, and then walk off to lunch, leaving the record locked the entire time. >.<

We've learned a lot about locks over the years.

 

[alank2]

You are all bringing up exactly the examples I am thinking about. One action might be that an application just needs to grab the latest data (a read), but another action might be that an application needs to grab the latest data, update it, and write it back. In this case, both start out with reads, but one read isn't so important (it gets old data, it gets new data, that data only might be displayed, etc., but it isn't being changed based on it), but the other read is critical, it has to be correct so that the subsequent write is also correct. Would the second case be where a lock was placed on the record being read/changed BEFORE it is read? That way when the 2nd process reads it, it can know for sure that another process won't be trying to change it?

 

[whartung]

Would the second case be where a lock was placed on the record being read/changed BEFORE it is read?

Yes, you have to lock, and read again, then update.

That's one of the reasons "Sally" had the record locked through lunch. A lot of systems, if you were looking at a record on the screen, it was no big deal.

But if you entered "Change mode", the system would lock the record the entire time the person was staring at the screen (or at lunch) because the intent was that a change was coming, and if someone (or something) else came in a made a change, it would quite likely be stomped on.

Consider this simple scenario, the Customer record has a field "total sales" (It's a contrived example, so...).

Sally opens up and starts changing the customers address for some reason. Meanwhile, Paul is running a posting process for new orders. Sally has the record with a sales number of $100, Pauls process reads that, add $10 to it, and stores it back. This was done in the blink of an eye. Meanwhile, Sally, updates the address, checks it against her form, says "This is good...Oh Hi Betty! How's Sam?...oh...1 sec", and saves the record -- with the $100 sales amount.

Now, if the program locks the record, Pauls process just sits there, trying to do the update, but can't because of the locked record. Sally finally updates, and Pauls process finishes, and everything is ok. Paul is just annoyed "Why is this taking so long". (Sally's not a bad person, computers are just awful, demanding monsters.)

Back in the day, it was not uncommon for the entire record to be loaded into the program, changed one part of it, and written back wholesale. Nowadays, things like SQL let systems be more selective (for example, the program Sally was using would simply not update the sales number field). And, of course, folks tend to separate things out more, data is more granular.

This type of hard locking is called "pessimistic" locking. Many systems for interactive use use a technique called "optimistic" locking -- which really isn't a lock at all. Rather, when they try to perform an update, they may qualify it with what their update does.

For example, in SQL, someone might want to change the address line from 100 Main Street to 101 Main Street.

They can try something like this: "UPDATE Customer SET Address='101 Main Street' WHERE Address = '100 Main Street'"

This will, obviously, lock the records for a microsecond to perform the update. But if someone else snuck in and made a change behind the back of this person, the update would simply fail. Then it's a matter of how the UI deals with that. (Nothing like filling out a bunch of changes and have the machine come back "So sorry, something changed" and "refresh" all of your fields on your screen. "Thanks a lot!")

There's all sorts of ways to manage that. But it just demonstrates the complexities of the problem, and how, over time, things have changed to manage it.

 

[alank2]

Thanks for the detailed reply whartung! I was testing the DOS open API command last night and noticed that they added bits to specify the sharing mode (similar to how WIN32 CreateFile allows you to specify if someone else can open the file and if so in what way). What I noticed about DOS is that it is only relevant/effective when SHARE.EXE is loaded. I wondered if this extended to the network sharing that DOS was familiar with so I setup a virtual machine running old Netware 4.1 to see what it would do. It does indeed enforce the lock bits in DOS open API, so the same file can be sopen()'d in the same process, or from 2 clients either. One odd thing though is that the Netware server console shows the file in use, but it doesn't show any locking done on it. I even tried to use the C lock() function to lock part of all of the file, and still the Netware console didn't show it as locked so maybe that is a Netware specific type of lock or something. None of the Netware testing required share to be loaded.

 

[Svenska]

Simple DOS applications do not require file locking at all (as long as they are single-threaded), and because memory was scarce and expensive, file locking is not implemented in DOS itself. The whole locking API only became important with network drives, and that support is implemented by the redirector. SHARE.EXE implements the API for local drives only. Applications needing functioning file locking therefore need to check first whether file locking is available on a particular drive.

A consequence of most DOS apps being blissfully unaware of file locking is that locks must be enforced by the system and default to the most pessimistic locking (exclusive access, whole file) by default to prevent most accidental data corruption. Windows inherited this locking style from DOS for compatibility reasons and built on top of that. Netware servers needs functional file locking on local drives for sharing, so they might as well have it always enforced.

UNIX systems traditionally do not enforce file locking at all. Applications not using the file locking API will also ignore any existing locks (of course, SysV and BSD systems used different APIs - flock vs lockf). However, this avoids the runtime open/access overhead experienced by DOS/Windows applications when file locking is not necessary. Also, both APIs have pitfalls, so many applications relied on file system guarantees and used lockfiles to lock out other applications.

 

[whartung]

Windows inherited this locking style from DOS for compatibility reasons and built on top of that.

A primary reason I do not like Windows is that you can't delete an open file. The files are locked on open, and deletes just fail. I seem to constantly bump into that back when I was doing more work on Windows. I don't now, and I don't miss it at all.