Evaluating Digital Libraries

Evaluating digital libraries, as a prototype, was conducted to fulfill the general requirement for International Open Public Digital Library (IOPDL).

Purpose

The main purpose and goal of evaluating Digital Libraries is to determine well-designed digital libraries (WDDLs). In here, Well-designed libraries (WDDLs) are defined as qualified libraries by usability, performance, and content evaluations in their subject domains. The determined WDDLs will consist of International Open Public Digital Library (IOPDL) along with its own new published collections. It is to fulfill the general requirement to establish IOPDL. Also, it is, generally, to find strength and weakness of existing digital libraries. The finding may enhance their strength and complement their weakness. It will improve overall quality of digital libraries.

Methodology

Through investigations of existing evaluation tools and methods, the main finding is that there is no appropriate method to evaluate several performances together. Existing methods evaluate one or two specific area(s) of many performances. Thus, one combined method is suggested to evaluate multiple performances together. The suggested evaluation criteria: Content, Usability and Performance Evaluation (CUPE) criteria. With CUPE criteria, mainly three evaluations are conducted for existing sixty three digital libraries: content evaluation, usability evaluation, and performance evaluation. The evaluation is limited regionally almost in the USA and timely on 2010.

What should we evaluate?

We should evaluate content of collections, their interfaces’ usability and performance of existing digital libraries, because these are requirements of well-designed digital libraries. Well-Designed Digital Libraries are the libraries that provide high quality content collections in a subject area(s), high quality usability and performance. WDDLs should satisfy the following requirements:

● WDDLs should have their own unique high-quality collections in a special subject area(s);
● They should provide convenient services in usability for users to use easily them;
● They should provide sufficient performance for users to access and use them without delaying time and with reliable.

Who should evaluate?

The evaluation was processed at Graduate School of Library and Information Science (GSLIS) in the University of Illinois at Urbana-Champaign in 2010. The evaluation reflects many discussions and instructions of Professor McDonough, Dean Smith, and Dean Renear of GSLIS. Especially, the content evaluation was conducted mainly by Professor McDonough. Based on instructions of the faculty, Boaz Sunyoung Jin evaluated existing digital libraries in respect to their content, usability and performance.

Upon what criteria do we evaluate?

The new suggested Content, Usability and Performance Evaluation (CUPE) criteria is used, which is a set of criteria in main Content, Usability and Performance Evaluation criteria. In the main three criteria, seven criteria and their checklists are included as follows. The criteria in the set are chosen from existing evaluation criteria in order to be appropriate for the purpose of the project.

1. Content quality evaluation criteria

– Accuracy – whether collections have accurate information in the subject area that the users can trust;
– Coverage – adequacy of the scope of the collection, considering both breadth and depth;
– Authority – how authoritative the site appears to be, based on the reputation of the
organization or sponsors; and
– Satisfaction – experts’ overall response to the digital library’s collection.

● The Usability Evaluation Criteria

2. Accessibility

Evaluate whether users can access information of a digital library with no or at least limitations without errors;

Interface Usability:
3. Convenience/ ease of use

Evaluate whether it is easy for users to learn and use the digital library in browsing and scanning (how much easy it is to navigate to from most pages);

4. Interfaces’ consistency

Evaluate whether it uses the same words, color, font, graphics, and layout among web pages;

5. Visible design and Aesthetic Appeal of a digital library

Evaluate whether it communicates clearly and visibly the purpose and value of the interface’s components with unique and descriptive headings and visibility, and whether it has attractive aesthetic appeal.

● The Performance Evaluation Criteria

6. Response, retrieval time

Evaluate how much time does it take to carry out tasks (navigate, browse, search, or obtain resources); the average time that a digital library takes to process all requests including the link response time and the search response time;

7. Relevance of obtained results (effectiveness, efficiency, and usefulness)

Evaluate how precise obtained results are from requested queries of users.

How to apply them in evaluation? (How will the evaluation of existing digital libraries be performed?)

With the CUPE criteria, several existing evaluation methods are used, such as heuristic, using open resources, and using specially designed computer programs. To check whether existing digital libraries satisfy the requirements of well-designed digital libraries, three steps are designed to evaluate existing digital libraries:

●The first step is Content evaluation that experts investigate the content of collections of existing digital libraries whether they have high quality collections in a particular subject area(s) using a set of content evaluation sub-criteria;
● The second step is interfaces’ usability evaluation that checks whether they provide high quality usability for users to use it without troubles and difficulties, using open tools and heuristic methodology, and using a set of usability evaluation sub-criteria;
● The third step is performance evaluation that checks whether they provide enough
performances for users to use it without delay and with reliability by simulations with specially designed computer programs, using a set of performance evaluation sub-criteria.

●Generally, 5-point rating scale applies to evaluate the candidate digital libraries with seven criteria of CUPE criteria. Depending on each criterion, 5-point scale applies to differently.

Total Results and Analyses

As a result, the total average of all candidate digital libraries is 3.2003 in 5-point scale. It may mean generally that the candidate digital libraries may provide enough quality usability and performance. They are recommended by content evaluation in their subject domains. Ultimately, thirty four digital libraries out of sixty two digital libraries show higher final averages than the total average, 3.2003. That is, about half of the candidate digital libraries are turned out as well-designed digital libraries in their subject domains.

NASA’s Visible Earth (4.452381) and Census Atlas of the United States (4.263889) in Geography subject area show the highest averages by seven evaluations with the CUPE criteria. Third, U.S. Department of Health & Human Services shows 4.261905 in medicine subject domain. Fourth, GPO Access in political science and law shows 4.142857. Fifth, National Science Foundation in science shows 4.138889. Total Results and Analyses web page shows more detail results based on scores.

Finally, Boaz Sunyoung Jin concludes that the U.S. has strongly specialized digital libraries in Geography subject area (4.1316), Medicine (3.5571), Military Science (3.496), Political Science and Law (3.4524), and Science (3.4454).

The turn out Well-Designed Digital Libraries by Evaluations

Summary of Conducted Seven Evaluations

The International Open Public Digital Library (IOPDL) is proposed as a global national digital library for the public and even the disabled to access any information in various subject areas. The proposed IOPDL will consist of well-designed digital libraries in fifteen subject areas by cooperation with them. The way will reduce a huge amount of budget saving much effort and time to establish it. The IOPDL can be used as an education community that provides life-long learning opportunities for the disabled and for all age with cutting-edge technologies.

To establish the proposed International Open Public Digital Library (IOPDL), it is necessary to find which digital libraries are well-designed digital libraries. To find out well-designed digital libraries, Content, Usability, and Performance Evaluation (CUPE) criteria is suggested to evaluate multiple performances of existing digital libraries. To evaluate existing digital libraries and to evaluate performance of the suggested CUPE, several evaluation methods are selected based on evaluation criteria.

By seven evaluations with the CUPE criteria, 34 well-designed digital libraries are selected among 62 candidate digital libraries in fifteen subject domains.

First, through the content quality evaluation, sixty three digital libraries are recommended as candidates of well-designed digital libraries in fifteen subject areas. The Content quality evaluation was done by faculty and Boaz Sunyoung Jin in Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign. We investigated Contents’ accuracy, authority, and scope of collections of existing digital libraries based on their subject domains.
Second, to evaluate Accessibility of sixty two candidate digital libraries, seven Accessibility Evaluation Tools are chosen based on various Accessibility standards. The method for evaluating accessibility gave enough credible results.
Further, to evaluate interfaces’ Usability, the heuristic evaluation method was selected. The author evaluated Interfaces’ Usability with Convenience, Interfaces’ Consistency, and Visible Design and Aesthetic Appeal Evaluation Criteria. Each criterion has their unique check lists that enable enough assessments in various points of views. Through the evaluations, general interfaces’ usability of the candidate digital libraries are evaluated.
To evaluate Performance, two computer programs are designed and executed. Through the programs, accurate results of response times and relevance rates can be easily achieved in a short time rather than using other evaluation methods.
Finally, the final average of each digital library is compared with the total average of sixty two digital libraries to decide which digital library can be well-designed digital libraries.

As a result, thirty four digital libraries show higher final averages than the total average. They turn out as well-designed digital libraries with enough evidences in Content, Usability, and Performance.

Discussion

Successful and Useful Suggested Content, Usability and Performance Evluation (CUPE) Criteria

Consequentially, seven evaluations with the CUPE criteria are very effective and efficiency to find well-designed digital libraries, and to evaluate multiple performances of many digital libraries together. Moreover, the suggested CUPE criteria are suitable to evaluate Contents’ quality of collection, Interfaces’ Usability, and Performance for sixty two candidate digital libraries. The selected thirty four well-designed digital libraries in fifteen subject areas may consist of the International Open Public Digital Library. It will be a cornerstone to establish the proposed IOPDL.

For Future Evaluations

The experiment is mainly done with the U.S. digital libraries except few other country libraries such as The National Archives, Education Resources UK, British Library Online Gallery, International children’s Digital Library, Chinese Philosophical Etext archive. However, for the International Open Public Digital Library (IOPDL), we should include and evaluate more national digital libraries of many countries.

Moreover, there are many good digital libraries that are not included and evaluated in the prototype. With other criteria, the digital libraries that are not turned out as WDDLs in the prototype can be Well-Designed digital libraries.

For the future content quality evaluation,

it is strongly recommended that many experts of subject domains be involved in evaluating Content quality of existing digital libraries. It will increase possibility to find more many high quality content digital libraries in many subject areas.
For the future Interface usability evaluation, it is desired that more users are involved in the evaluation. It might give better objective majority assessments for Interface Usability evaluation. But, it is also true that users cannot evaluate many digital libraries with the same criteria without direct rewards.
In the performance evaluation, synonyms and combination words as queries are not implemented to evaluate response time and relevance of query results. If synonyms and combination words are considered as queries, we can increase performance of search engines.

Solving Interoperability Problem

For the next step, therefore, it is necessary for thirty four well-designed digital libraries to cooperate solving interoperability problems. Further researches are required to achieve practically interoperability among well-designed digital libraries. For it, another prototype is planned to build interoperability among three universities libraries with a Common Terminology. A Common Terminology can be designed with several metadata schemas (e.g., MARC, MODS, and DC & QDC). If we can achieve practically interoperability among universities’ digital libraries with the Common Terminology, the achieved interoperability method can be expanded and be generalized to achieve interoperability among well-designed digital libraries. Ultimately, through the generalized interoperability method, the proposed International Open Public Digital Library can be established with cooperation with well-designed digital libraries.

Paper

Evaluating Existing Digital Libraries as a Prototype with the Suggested Criteria: Content, Usability, and Performance Evaluation Criteria, available on https://www.ideals.illinois.edu/handle/2142/50102.

Reference

About /robots.txt. (2007). Retrieved from About /robots.txt : http://www.robotstxt.org/robotstxt.html

Arms, W. (2000). Digital libraries. MIT Press. Retrieved from http://books.google.com/books?id=pzmt3pcBuGYC&printsec=frontcover&source=gbs_navlinks_s#v=onepage&q=&f=false

Bishop, A. P. (1998). Logins and bailouts: Measuring access, use, and success in digital libraries. The Journal of Electronic Publishing, 4(no.2). Retrieved from http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0004.207

Borgman. (2003). At least four types of evaluation that are related to digital libraries. NSF Report on DELOS Workshop on DL Evaluation, 6. Retrieved from http://www.scribd.com/doc/995922/National-Science-Foundation-evaluation

Brajnik, G. (2008). A comparative test of web accessibility evaluation methods. . ACM SIGACCESS Conference on Computers and Accessibility, 113-120.

CIMT. (n.d.). Unit 5 Section 2 : Mean, Median, Mode and Range. Retrieved from Centre for Innovation in Mathematics Teaching: http://www.cimt.plymouth.ac.uk/projects/mepres/book8/bk8i5/bk8_5i2.htm

CybthiaSays. (n.d.). HiSoftware® cynthia says™ portal – web content accessibility report. Retrieved from CybthiaSays: http://www.cynthiasays.com/

Etre. (n.d.). Etre accessibility chec. Retrieved from http://www.etre.com/tools/accessibilitycheck/

Fuhr, N., Tsakonas, G., & Aalberg, T. (2007). Evaluation of digital libraries. Springer-Verlag.

Fujitsu. (n.d.). Fujitsu web accessibility inspector 5.11. Retrieved from http://www.fujitsu.com/global/accessibility/assistance/wi/

Hilbert, D. M., & Redmiles, D. F. (2000). Extracting usability information from user interface events – university of california at irvine. ACM Computing Surveys, 32(No. 4), 384-421.

IITAA. (n.d.). Illinois information technology accessibility act standards 1.0. Retrieved from http://www.dhs.state.il.us/IITAA/IITAAStandards.html

ISO. (n.d.). International Organization for Standardization. Retrieved 2010, from http://www.iso.org/iso

Jin, B. S. (2014). International Open Public Digital Library (IOPDL): A Proposal for the Future. Illinois Digital Environment for Access to Learning and Scholarship (IDEALS). Retrieved from http://hdl.handle.net/2142/50101

Johnson, M. (2008). Evaluation of a digital library implementation UBC library digital collections and services: Digital collections. FIS2010 Digital Libraries.

Nielsen, J. (1992). Finding Usability Problems Through heuristic Evaluation. ACM CHI ’92.

Nielsen, J. (1993). Response Times: The 3 Important Limits. Retrieved from http://www.nngroup.com/articles/response-times-3-important-limits/

Nielsen, J. (1993). Usability engineering. San Francisco: Morgan Kaufmann.

Nielsen, J. (1995). How to Conduct a Heuristic Evaluation. Retrieved from Nielsen Norman Group: http://www.nngroup.com/articles/how-to-conduct-a-heuristic-evaluation/

Nielsen, J. (2008, February 19). Top 10 Application-Design Mistakes. Retrieved from http://www.useit.com/alertbox/application-mistakes.html

Nielsen, J. (2011). Top 10 Mistakes in Web Design. Retrieved from http://www.useit.com/alertbox/9605.html

Reeves, A. X., & Woo, Y. H. (2003). Evaluating Digital Libraries: A Userfriendly Guide. Digital Library of Information Science& Technology.

Saracevic, T. (2000). Digital Library Evaluation: Toward an Evolution of Concepts. Library Trends, 49(3), 350-369. Retrieved Feb. 15, 2010, from http://comminfo.rutgers.edu/~tefko/LibraryTrends2000.pdf

Saracevic, T. (2004, September). Evaluation of digital libraries: An overview. Retrieved from http://comminfo.rutgers.edu/~tefko/DL_evaluation_Delos.pdf

SEO. (n.d.). SEO Chat Keyword Density Tool – SEO Tool. Retrieved from http://seo-software.findthebest.com/l/82/SEO-Chat-Keyword-Density-Tool

SEO Tools – Keyword Density. (n.d.). Retrieved from javamazon.com: http://www.javamazon.com/seo-tools-keyword-density/

Tammaro, A. M., & Marlino, M. (n.d.). Digital Library Evaluation. Retrieved from http://www.delos.info/files/pdf/DELOS_NSDL_sschool_07/Presentations/Marlino_Tammaro.pdf

UIUC. (n.d.). Functional accessibility evaluator 1.0.3. Retrieved from University of Illinois at Urbana-Champaign: http://fae.cita.uiuc.edu/

Usability.gov. (n.d.). Usability.gov: Guidelines. Retrieved 2010, from http://www.usability.gov/guidelines/

W3C. (n.d.). Improving the Accessibility of Your Website. Retrieved from Web Accessibility Initiative: http://www.w3.org/WAI/impl/improving

W3C. (n.d.). W3C Markup Validation Service. Retrieved from http://validator.w3.org/

WAEX. (n.d.). Web Accessibility Evaluator in a single XSLT file (WAEX). Retrieved from http://www.it.uc3m.es/vlc/waex.html

WAVE. (n.d.). Web Accessibility Evaluation Tool (WAVE). Retrieved from http://wave.webaim.org/

webconfs.com. (n.d.). What is Robots.txt. Retrieved from webconfs.com: http://www.webconfs.com/what-is-robots-txt-article-12.php

Weisberg, H. (1992). Central Tendency and Variability. SAGE, 83 .

Xie, H. (2006). Evaluation of digital libraries: Criteria and problems from users’ perspectives. Library &Information Science Research, 28, 433-452.

Zhang, D., & Si, L. (2009). Modeling search response time. ACM.

Evaluating Digital Libraries