Skip to main content

Text and Data Mining Library Databases: Policies for Library Subscription Databases

Guidance for text and data mining subscription resources

Contact Information

Please contact Jen Ferguson, Research Data Management Librarian, with any questions:

j.ferguson@neu.edu / 617-373-3853

Library Support in a Nutshell

We can provide:

  • This research guide with resources, contact information, other details
  • Consultation with librarians
  • Annual payment for membership in HathiTrust
  • Negotiation with vendors to include general text mining provision in licenses for library resources

Unfortunately, we cannot generally provide:

  • Additional payments for data to enable text mining -- the library is not funded for this activity
  • Secure storage for vendor data (this is sometimes possible, but depends on the case, so we cannot guarantee it)
  • Licensing for individual text mining projects -- generally, the researcher needs to negotiate the license directly with the vendor, unless the vendor requests addendum to the library-wide license
  • Guarantees on enforcing user behavior and handling of vendor data

Tell us!

Are you working on a text mining project?  Do you have feedback on your experiences working with library vendors?  Contact us - we'd love to hear about it!

Text & Data Mining of Library Subscription Databases

This site is intended to support to members of the Northeastern University community interested in text/data mining library-provided subscription resources.  

For researchers interested in using the Libraries’ subscription content for the purposes of text or data mining, here are some things you should know. The process can sometimes be a little arcane, so if you have any questions, please contact us -- we are here to help!

  1. You may need a librarian’s assistance. Many publishers and vendors such as Thomson Reuters, EBSCO, and ProQuest restrict automated data scraping and large­ scale access to their journal articles and databases. They believe doing so is necessary to protect their copyright, so that end ­users do not abuse their subscription privileges by mass downloading and sharing with others who have not paid for a subscription. They also want to avoid heavy loads on their systems. We can help you contact publishers in order to gain access -- email Amanda Rust (a.rust@neu.edu) to get started.

  2. It might take some time. Text and data mining is sometimes negotiated with publishers on a case-­by­-case basis. Other University offices may be involved, and it may take weeks or months to work out terms with vendors. Some cases go relatively quickly, but many do not. Be sure to build this extra time into your research schedule.

  3. Negotiations with publishers are mostly handled by the researcher. Librarians can help you make initial contact with vendors, but sometimes the responsibility to communicate with the vendor and acquire the data falls to you, the researcher. Depending on the contract, if you are asked to represent or be liable for the University, you will want to consult with the University Office of General Counsel. In some cases, vendors may ask for an addendum to the existing library license -- we will happily work with you in those cases. 

  4. It might cost you money (and resources). Only a few of our current vendor contracts include text­ or data­ mining access as part of the general license, and there is often a data preparation fee, though we are advocating on researchers' behalf to wherever possible include text and data mining access into our licenses with minimal additional fees. However, depending on the vendor, one time time access can cost thousands of dollars. Writing expected costs into grant proposals is a good way to ensure access.

  5. You will need to have your own data storage provisions and tools. Depending on the license and level of security needed, the Library's Digital Repository Service may be available. If the license requires high security, you may need to establish access to the University's High Performance Computing Center.  

  6. For certain types of text­ mining research, Open Access journals and repositories can be good alternatives. Publishers such as Hindawi, PLoS, and BioMed Central welcome text­ mining and reuse of their content, as do some institutional and subject repositories such as PubMed Central.

We are continuing to advocate for the inclusion of text mining rights in our licenses, so if there are particular vendors in which you are interested, please let us know.

Last updated October 2015. Adapted from text used by Indiana University - Bloomington Libraries, with thanks to Stacy Konkiel for providing the text.

Word of Advice

Generally, it's a bad idea to use automated means to scrape or download large amounts of data from any database to which the library subcribes. If a database provider allows text mining, they will want to provide the data for you in a secure manner with which they are comfortable.