Objectivism Research CD, How was it produced? Options

[R Wray]

There is an advertisement in the Forum Capitalism Corner Marketplace for the Objectivism Research CD-ROM. I am not that computer/scanner literate, but it seems to me that this CD is an amazing piece of technology. I have found it very reliable and user friendly.

I am very interested to know the basics of how the CD was produced. Were the source materials in bound form? If so, how were the pages turned for the scanner? I assume the images are converted to digital character data. What is the error rate; does each line have to proofed by a human being? (In using the CD for two years or more, I think I found only two small typos that might have been caused by flyspecks.)

I’m wondering how long the whole process took.

(I understand that complete libraries are/will be digitized for the Internet. Is this by the same process?)

[PhilO]

I can answer this, since I was the developer of the Objectivism Research CD-ROM.

Essentially, I scanned the material in electronically from standard material that had been debound. I did most of the scanning on a Fujitsu M3099A high speed duplex scanner, a device that can rapidly scan both sides of each page from a pile of pages. The material was then converted to raw text using an optical character recognition program called Typereader, which at the time was the best one available, as far as I could find.

The OCR conversion process wasn't perfect and required manual cleanup. The OCR program provided a way to systematically traverse a set of scanned pages, giving the opportunity to fix letters/words that the software wasn't "sure" about, as well as words that were not in the internal dictionary. After that initial cleanup, the text was output into rich text format for consolidation into Microsoft Word files, where I did additional spell checking and cleanup. The conversion process did occasionally make mistakes, the worst ones being where the program guessed at a word incorrectly and substituted a valid but wrong word, which was missed by later spell checking. Fortunately that was relatively rare.

After all of that, I applied internal markup to the Word files to systematically break up the documents into logical subdivisions such as chapters or articles, with appropriate titles. Then a commercial third party program was used to index that material to create the final textual database that is used for the browsing and fulltext searches. The end-user interface is a program that I wrote using Microsoft C++/MFC. The markup provides a way to browse or see fulltext results at a high level. When you do a browse or a search, the article/chapter titles that you see in the lefthand window come from that markup.

I didn't keep track of how long the entire process took overall, unfortunately, because it was a part-time effort in between "real" work. Initially it was only for myself, with an eye towards possible licensing of the material to make a product. As it turns out, getting the licensing took far longer than the actual project (years longer), a somewhat frustrating process. It was worth it though - I think Ayn Rand's ideas are extremely important and I enjoy selling a product that helps people to read and study them in a coherent way.

I do keep track of typos found and apply corrections that eventually make it back into the product - anybody finding one can let me know at: typos AT objectivism DOT net.

As a little advertisement: expect to see The Intellectual Activist for the years edited by Peter Schwartz available in a bound collection, as well as electronically searchable form, this year. The project is close to completion.

I'll comment on the last part of the original question in a separate post.

-- Phil Oliver (www.Objectivism.net)

[PhilO]

Regarding the mass conversion of libraries to digital form - it's my understanding that Google has plans, in cooperation with some of the major public and university libraries of the world, to scan in millions of books for a massive searchable database. From what I've been able to find out about the effort, they plan to use costly book scanning devices that can non-destructively scan in pages and automatically turn the pages - clearly the only kind of scanning that would be both practical and acceptable, in this context. The main non-technical issue facing Google are copyrights on non-public domain works.

Amazon.com has some experience in this area, with their "search within the book" system; they've had many thousands of copyrighted books scanned into digital form and made fulltext searchable by customers. The differences between this and the Google effort are:

(1) The scope of the projected project - Google is talking about digitizing entire large libraries.
(2) Amazon's interest is primarily selling books. Search within the Book doesn't let anybody read the entire work online, just the pages with the search hit +/- a few pages (as I recall.)
(3) Amazon had to get explicit permission from each publisher for all of the books that it digitized. That's not really possible in the case of Google's effort, given the vast scope of material involved. It is not always easy to find out if a work is even in the public domain, and if it is not, who owns the rights, and which rights they own. The only straightforward way to tell if a work is in the public domain is by publication date, per this:

Cornell University's page on Copyright Term and Public Domain in the U.S.

In a nutshell, any work published before 1923 is now in the public domain. There are many other works after that time which are also in the public domain - and there are large numbers of works which will not be in the public domain for many decades.

So, although it's an exciting project, both technically and in terms of the final results, I think they'll run into legal issues that will tie it up for a long time, unless they choose to focus only on works which are definitely public domain.

[R Wray]

Phil,

Thanks very much for your illuminating posts. Sounds like some of the work on the CD data was very tedious, and the user interface program was probably no piece of cake. After all that hard work, it’s regretful that the licensing was such a problem. I’m glad that you persevered—great job.

[PhilO]

Is it going out of print? No new edition with all the new material?

I don't plan to go through another irrational struggle to prolong the licensing. My original naive view, years ago, was that Leonard Peikoff would understand the value of the CD-ROM, but he does not see it as a value and evidently has other priorities these days than seeing the ideas of Ayn Rand spread as widely as possible, with the greatest level of accessibility.

Are there copies preserved in acid-free plastic safely hidden in caves?

Probably meant as a joke, but to answer you seriously, not by me. However, for several years I've used the costly Mitsui "gold" CD-Rs, which have an actual gold (non-oxidizing) reflective layer as well as much more stable dyes, which will have a far longer lifetime than other CD/CD-Rs - exactly because I want them around for a long time, perhaps past a near term collapse of the world. As digital artifacts it would be tougher to retrieve the data than simple physical printing, but at least they would exist, and in a form used by hundreds of billions of other such artifacts (CD/DVD discs), which is a motivator for future generations to have the technology to read the media.

[ewv]

Is it going out of print? No new edition with all the new material?

I don't plan to go through another irrational struggle to prolong the licensing. My original naive view, years ago, was that Leonard Peikoff would understand the value of the CD-ROM, but he does not see it as a value and evidently has other priorities these days than seeing the ideas of Ayn Rand spread as widely as possible, with the greatest level of accessibility.

Do you mean the original issue or a previous extension was a problem? Can you tell us what the objections to it were? I think it is clear to everyone here that what you did was of enormous value; at least I have never heard an objection. Apparently ARI thinks so too because they have been selling it (or at least used to, are they out of them?).

Are there copies preserved in acid-free plastic safely hidden in caves?

Probably meant as a joke, but to answer you seriously, not by me. However, for several years I've used the costly Mitsui "gold" CD-Rs, which have an actual gold (non-oxidizing) reflective layer as well as much more stable dyes, which will have a far longer lifetime than other CD/CD-Rs - exactly because I want them around for a long time, perhaps past a near term collapse of the world. As digital artifacts it would be tougher to retrieve the data than simple physical printing, but at least they would exist, and in a form used by hundreds of billions of other such artifacts (CD/DVD discs), which is a motivator for future generations to have the technology to read the media.

The best DVD's I know of for archiving are Taiyo Yuden DVD+R, which I use for all backups now. Is that what you recommend or is there a gold Mitsui version of DVDs that is better? I wonder if the billions of CDs will be enough to preserve the technology for reading the format. How long do most of them last? Devices for older formats like VHS, cassettes, 1/4" tape, records, wire recordings and wax cylinders are getting harder to find all the time, at least for consumers, and the old tapes especially are flaking apart anyway. (I have some originals of Ayn Rand at Ford Hall back to the late 1960's that I "inherited" from some defunct group that passed them along.) I wonder if it will be expected that all valuable CDs and DVDs be transferred to something else before they go bad, with a corresponding lesser need for the technology in the future for reading copies of Phil's CDs found in caves. I keep a copy on my hard drive for easier access, so it is routinely backed up repeatedly and is archived monthly along with everything else, but I have been considering ripping multiple copies and shipping them to Africa for safe keeping . When aliens come here seeking to revive our collapsed world, that stash may be the most recent still readable source.

Anyway, I hope your CD is continued and expanded some time in the future, and highly recommend that anyone on the Forum who doesn't already have it get it while they can. I think that almost all books will probably be in digital form some day, a project that is already underway.

(Maybe Betsy will want to abort this discussion to a new thread. )

[PhilO]

Do you mean the original issue or a previous extension was a problem? Can you tell us what the objections to it were? I think it is clear to everyone here that what you did was of enormous value; at least I have never heard an objection. Apparently ARI thinks so too because they have been selling it (or at least used to, are they out of them?).

It's not really something I want to go into in detail currently, but to slightly elaborate, I mean that he, personally, does not see the value of the ROM - as opposed to ARI and other Objectivist intellectuals.

The best DVD's I know of for archiving are Taiyo Yuden DVD+R, which I use for all backups now. Is that what you recommend or is there a gold Mitsui version of DVDs that is better?

I'm not personally familiar with that brand, but note that DVD media is significantly different than CD media because of differences in the physical format (higher density) and differences in read/write lasers (I think.) Mitsui's estimated lifetime of their gold CD-R media is 200-300 years, but barring physical damage and leaving them out in direct sunlight, I would guess far longer than that.

I wonder if the billions of CDs will be enough to preserve the technology for reading the format. How long do most of them last? Devices for older formats like VHS, cassettes, 1/4" tape, records, wire recordings and wax cylinders are getting harder to find all the time, at least for consumers, and the old tapes especially are flaking apart anyway.

A good point, but I personally think that the situation with CD/DVDs is qualitatively different. 1) Sheer quantity, which I believe far exceed any previous media format. 2) Digital format. 3) Physically more robust than magnetic media, which can be slowly erased by natural magnetic fields (e.g. the earth) as well as being physically more fragile and prone to deterioration. 4) Extremely large storage capacity for their size, especially relative to prior forms of media, means that vast amounts of audio/visual/computer data are now archived in that format, further increasing the motivation for future generations to read them. 5) Billions of drives are available to read these standard formats, far exceeding the number of e.g. magnetic tape readers in the 60's. 6) There is redundant data on the discs which permits a certain number of bits to become unreadable.

Note also that future technology to read today's media need not be like today's drives. It may not be many more years before ultra resolution cameras could simply take a "snapshot" of the surface of the disc and then a computer can apply image processing to recover the bitstream. I would guess that approach would permit future reading of the media almost instantaneously.

I keep a copy on my hard drive for easier access, so it is routinely backed up repeatedly and is archived monthly along with everything else, but I have been considering ripping multiple copies and shipping them to Africa for safe keeping . When aliens come here seeking to revive our collapsed world, that stash may be the most recent still readable source.

Back when I was working on a database for a gas turbine engine company to store properties of high temperature superalloys, I had the idea of using thin sheets of them, imprinted at reduced scale with actual page images of Ayn Rand's works (plus additional critical references for a future civilization that took staggering numbers of man-years to discover, e.g. CRC Handbooks, etc.) The imprinting could be done with lasers or etching with some scary acid solutions. With that approach the material can be recovered with pretty low tech means (magnifying glass) and is far harder to destroy than paper (or for that matter, CDs), especially by other low tech means - in other words, relatively immune to brutes who burn books but recoverable by a future thinking man.

Anyway, I hope your CD is continued and expanded some time in the future, and highly recommend that anyone on the Forum who doesn't already have it get it while they can. I think that almost all books will probably be in digital form some day, a project that is already underway.

Well, someday the material will all be out of copyright...

Yeah, Google has been funding that effort. It's cool in one way, but they're also being very cavalier about copyrights, and the publishers of in-copyright books are not happy about it.

[ewv]

It's not really something I want to go into in detail currently, but to slightly elaborate, I mean that he, personally, does not see the value of the ROM - as opposed to ARI and other Objectivist intellectuals.

Was there some philosophical argument for rejecting its value or a more mundane dismissal? I ask because he tends to see everything philosophically.

The best DVD's I know of for archiving are Taiyo Yuden DVD+R, which I use for all backups now. Is that what you recommend or is there a gold Mitsui version of DVDs that is better?

I'm not personally familiar with that brand, but note that DVD media is significantly different than CD media because of differences in the physical format (higher density) and differences in read/write lasers (I think.) Mitsui's estimated lifetime of their gold CD-R media is 200-300 years, but barring physical damage and leaving them out in direct sunlight, I would guess far longer than that.

Do they have a version of that for DVD's? (I think the lasers are different because the wavelength of the light has to be shorter for the higher resolution.) Taiyo Yuden makes high quality discs consistently (other brands are sometimes good but not consistent across the manufacturers they use) but I don't think they use gold, at least not routinely.

I wonder if the billions of CDs will be enough to preserve the technology for reading the format. How long do most of them last? Devices for older formats like VHS, cassettes, 1/4" tape, records, wire recordings and wax cylinders are getting harder to find all the time, at least for consumers, and the old tapes especially are flaking apart anyway.

A good point, but I personally think that the situation with CD/DVDs is qualitatively different. 1) Sheer quantity, which I believe far exceed any previous media format. 2) Digital format. 3) Physically more robust than magnetic media, which can be slowly erased by natural magnetic fields (e.g. the earth) as well as being physically more fragile and prone to deterioration. 4) Extremely large storage capacity for their size, especially relative to prior forms of media, means that vast amounts of audio/visual/computer data are now archived in that format, further increasing the motivation for future generations to read them. 5) Billions of drives are available to read these standard formats, far exceeding the number of e.g. magnetic tape readers in the 60's. 6) There is redundant data on the discs which permits a certain number of bits to become unreadable.

Note also that future technology to read today's media need not be like today's drives. It may not be many more years before ultra resolution cameras could simply take a "snapshot" of the surface of the disc and then a computer can apply image processing to recover the bitstream. I would guess that approach would permit future reading of the media almost instantaneously.

Because of the high spacial resolution it would have to be scanned by a laser somehow; I don't think a literal snapshot getting the whole field at once would be possible with ordinary light. But I don't know anything about what someone may be working on to make the process faster with a different kind of scan and image processing. I don't think the billions of drives now out there are relevant because they won't last long enough even if the CDs and DVDs do. But I hope there is a better transistion in how the format is read this time which continues to make use of the current media. Digital tape also has redundancy and tape has the advantage of being easily recoverable even if there are some dropouts causing small actual losses; digital disks are hard to read once a small portion is lost. Maybe we can just ignore the wax cylinders.

Back when I was working on a database for a gas turbine engine company to store properties of high temperature superalloys, I had the idea of using thin sheets of them, imprinted at reduced scale with actual page images of Ayn Rand's works (plus additional critical references for a future civilization that took staggering numbers of man-years to discover, e.g. CRC Handbooks, etc.) The imprinting could be done with lasers or etching with some scary acid solutions. With that approach the material can be recovered with pretty low tech means (magnifying glass) and is far harder to destroy than paper (or for that matter, CDs), especially by other low tech means - in other words, relatively immune to brutes who burn books but recoverable by a future thinking man.

A precursor to nano-technology?

Anyway, I hope your CD is continued and expanded some time in the future, and highly recommend that anyone on the Forum who doesn't already have it get it while they can. I think that almost all books will probably be in digital form some day, a project that is already underway.

Well, someday the material will all be out of copyright...

Yeah, Google has been funding that effort. It's cool in one way, but they're also being very cavalier about copyrights, and the publishers of in-copyright books are not happy about it.

Major libraries like Harvard are also into it, presumably starting with out-of-copyright volumes now disintegrating, but even if they aren't, why not if it is only a backup archive for a library or a replacement of magazine formats taking up a lot of space in the library? I suppose the publishers would be less happy when multiple access is allowed and it is easy to do across university networks or beyond.

[Cometmaker]

The lower tech the better, so the media and the media reading device (and a power source) would not have to be stored to avoid excessive interdependence on devices with a short life cycle. I would like to see the emergence of this new market for superalloys with distributors and superalloy-book archivists and keywords and indices on the superalloys, too.

Microfilm hasn't worked out too handily. I would be willing to pay a small fee to a publisher through a library to keep backup archives for certain books, but this would be to get a premium member status above and beyond the archive-worthy material chosen by the library, a lot of which would be drivel, useful only for comparison purposes. The so-called premium member status I'm thinking of would allow access to other premium members' chosen backup archives (as well as the other institutions worldwide). Just like with research journals with contents available online, perhaps one would need to be a paying member to access content to make it somewhat worthwhile for publishers to integrate advertising and turn multiple access to their advantage.

I don't recall whether Harvard is using CSC Paper Saver for mass deacidification of its volumes, but my recollection is that this process prolongs a volume's physical integrity several centuries.

[ewv]

As a little advertisement: expect to see The Intellectual Activist for the years edited by Peter Schwartz available in a bound collection, as well as electronically searchable form, this year. The project is close to completion.

What is the status of this?

[PhilO]

Was there some philosophical argument for rejecting its value or a more mundane dismissal? I ask because he tends to see everything philosophically.

I think it's because he considers it irrelevant. The material is on paper, what more is needed?

Because of the high spacial resolution it would have to be scanned by a laser somehow; I don't think a literal snapshot getting the whole field at once would be possible with ordinary light. But I don't know anything about what someone may be working on to make the process faster with a different kind of scan and image processing.

I'd think that a very large sensor with trillions of sensor pixels could pick up the entire image of a CD or DVD, with appropriate optics. The "pits" on the discs are large enough to be picked up with visible/near infrared laser light. In any case, I still think it's likely that at least a few CD/DVD drives will survive intact in the future, as well as plans for building them or their higher speed successors, of the large number in existence now. The advantage of a "snapshot" approach is that a single robotic loader could rapidly recover the data from millions of discs.

Major libraries like Harvard are also into it, presumably starting with out-of-copyright volumes now disintegrating, but even if they aren't, why not if it is only a backup archive for a library or a replacement of magazine formats taking up a lot of space in the library? I suppose the publishers would be less happy when multiple access is allowed and it is easy to do across university networks or beyond.

The book publishers are dinosaurs (as are music publishers). Up until recently - and I surmise even right now - some of them actually re-key novels for reprints. Yes, re-key manually as opposed to saving a disk file. So, they do not even now really understand the power of digitizing for the internet. However, Google basically ignored the copyright issue rather than addressing it head-on and is creating a lot of bad will. It's given many universities the equipment to perform the scanning for them, and has unilaterally decided the "right" amount of copyrighted text that searches can show. There's no question that the technological means exists to digitize every book; it's been possible for many years. I can attest that the harder problem is the legal one. I think that Google believes that their size will permit them to ignore the problem, but losing a massive copyright infringement suit would destroy any corporation. I think it would be much more beneficial if Google and the publishers figured out an amicable way to accomplish the great goal of digitizing all human knowledge before the whole exercise gets undermined.

[softwareGuru]

... You might want to buy The Objectivism Research CD-ROM while it's still available.

I'm sorry to hear this. Will it be unavailable after a particular date, or after a certain number of copies are sold? I'm curious if I should be stocking up on some extra copies.

[PhilO]

I'm sorry to hear this. Will it be unavailable after a particular date, or after a certain number of copies are sold? I'm curious if I should be stocking up on some extra copies.

Until about March 1, 2008.

[softwareGuru]

Is there a restriction on resale? Purely hypothetically, if someone were to buy a 100 copies just before that date, would they be obliged not to re-sell after that date?

[PhilO]

Is there a restriction on resale? Purely hypothetically, if someone were to buy a 100 copies just before that date, would they be obliged not to re-sell after that date?

"I am not a lawyer", but it's my understanding that it is very well established copyright law that the owner of a copyrighted physical product has the right to re-sell that product - for example, nobody violates copyright law by buying a book then reselling it, though they would if they copied the book and sold the copies.

[ewv]

Was there some philosophical argument for rejecting its value or a more mundane dismissal? I ask because he tends to see everything philosophically.

I think it's because he considers it irrelevant. The material is on paper, what more is needed?

RTFS?

I'd think that a very large sensor with trillions of sensor pixels could pick up the entire image of a CD or DVD, with appropriate optics. The "pits" on the discs are large enough to be picked up with visible/near infrared laser light... The advantage of a "snapshot" approach is that a single robotic loader could rapidly recover the data from millions of discs.

I agree it would be a valuable technology. How do you get laser light reflected off the whole field at once? Lasers have a narrow beam. Has anyone tested a camera with anywhere near that resolution?

[PhilO]

RTFS?

I agree it would be a valuable technology. How do you get laser light reflected off the whole field at once? Lasers have a narrow beam. Has anyone tested a camera with anywhere near that resolution?

I should emphasize that I was positing some future technology, not anything available today. The trend is clearly towards escalating sensor density and sensitivity. I would not imagine that coherent (laser) light would be used to illuminate the disc; I'm envisioning simply bathing it in some evenly lit monochromatic light and taking the shot using reflected light and appropriate optical systems, literally imaging all of the pits on the disc then using fancy image processing to "virtually" read the bitstream. This is just imagination at the moment.

[Betsy Speicher]

Posts not directly relevant to the discussion of the CDROM have been moved to the "Response To Charges Against THE FORUM" thread here where it is more on-topic..

On that thread, in accordance with THE FORUM's policy against mind-reading, I have or will delete any posts speculating about anyone's motives. Criticism or analysis of a person's actual, citable, demonstrable actions and statements is fair game.

[ashley.parker.an...]

Another direction for preservation of text has occurred to me; I don't know whether anyone else has had or is pursuing the idea. Namely, why not also improve the technology for producing durable, even indestructible hard copy? One component of this might well be using a storage medium that would be made up of combined soft and hard copy--that is, the ultra-strong pages have their total digital equivalent embedded in each displayed letter.