old.kete.net.nz

Open Source Software for Community Digital Libraries

Conference paper written for My Language Conference held in Sydney, 10 - 11th August 2010.

Filename: RANSOM.doc ( download )

Size: 750.5 KB

Document type: application/msword ( Replace description with document text )

<!--Section Begins-->

Kete : Open Source Software for Community Digital Libraries.

Joann RANSOM

Head of Libraries

Horowhenua Library Trust

jransom@library.org.nz

http://old.kete.net.nz

Abstract:

Kete Horowhenua is a community built digital library of arts, cultural and heritage resources about the District in which I live. It aims to get the private collections, memories and knowledge of our community sitting alongside our public collections. Volunteers have contributed many thousands of hours to this project which has resulted in fully keyword searchable documents, digital images, audio and video clips, discussion and web links.

The software has since been released as Open Source under the GNU Public Licence (GPL) and other communities around the world are starting to fill their own Kete around any number of ‘themes’ or ‘subjects’.

Te Reo o Taranaki Charitable Trust is using Kete in community indigenous language revitalisation in Taranaki, Aotearoa, NZ. This paper serves as an introduction to the Kete project generally, and is a companion paper to Tuku Reo : Tuku Mouri : the Strategic Revitalisation of Taranaki Reo by Claire Hall.

Setting the Scene

Horowhenua Library Trust is a registered charitable trust established by the Horowhenua District Council in 1996. The Trust receives about 85% of its funding from Council and raises the remaining 15% from user charges, grants and fundraising projects.

While the Trust is a Council Controlled Organisation in terms of reporting, it is largely self-determining within the generous confines of its Trust Deed. Being outside the multi-layered government bureaucracy, the Trust is agile in its decision making. A result of its innovative approach to solving problems was the development of Koha, the world's first open source library management system, back in 2000. This product is now in use on every continent around the world and has been translated into many languages.

Horowhenua District Libraries serve a population of 30,000 of which approximately 18,000 are in the urban boundaries of Levin. Levin Library is the central library of the system, servicing 2 smaller branch libraries. The Trust employs 16 FTEs (full time equivalent staff) and many volunteers who total about 3 FTEs.

Horowhenua region is 1.5 hours north of Wellington, near the middle of New Zealand. The mild climate and rich soils make market gardening, horticulture and farming the mainstays of the economy. Manufacturing, food processing, construction and service industries are also important contributors. The population of the area has remained static for several decades and is projected to remain so.

Our District has high proportions of retired people with most of our children leaving the District at around 18 years for tertiary education and employment opportunities. Compared with the rest of New Zealand it is also statistically poorer and less educated. The people are predominantly European 72% with 22% identifying as Maori, 3% Asian and 3% Pacific Island. This is different to New Zealand as a whole and very different to our biggest population centre, Auckland.

1/3 of New Zealand’s population lives in Auckland which is also the most ethnically diverse region with 56% of its population identifying as European, 19% as Asian, 14% Pacific and 11% Maori. Middle Eastern, Latin American and African people combined make up less than 1% of New Zealand’s total population. 23% of our usual resident population was born overseas.

The Problem

Our Community had a problem.

The Library Trust has long worked closely with the local historical societies. We were keenly aware that these groups were struggling, with little professional expertise, few members and very little money. It was a challenge trying to balance the conflicting goals of enabling access to the District’s resources while preserving and protecting them for future generations.

Our District also has many small, specialised local museum and gallery-like organisations – and some not so small. Situated near our northern boundary, MAVTECH is the largest collection of audio and video technology in the southern hemisphere. Thousands and thousands of items – and none of it even catalogued! They wanted to do it but didn’t know where to start or even how.

As a known and trusted organisation in our community, the Library was looked to for guidance. We were asked by three different organisations, in as many months, to recommend a way forward for the heritage sector that would maximize precious resources: space, money, expertise and especially people.

The local Council was aware of the problems too. In 2004 the Library Trust carried out an audit of Arts, Culture and Heritage resources for Horowhenua District Council, to assess the extent of the resources currently held in the District and the long-term 'safety' of these resources for future generations. The findings were not surprising:

There is a large amount of material in private hands.

About half may be given to public collections – but half never will,

Most of it is available for loan or copying,

Lots of information is in people’s heads,

Everyone knew someone else with more material and knowledge,

People really do care about arts, cultural and heritage resources,

Physical space is a real issue,

A major concern was the lack of access to arts, cultural and heritage information.

The Library Trust then sat down and talked with a number of focus groups to clarify and confirm the problems we thought we had identified and to envisage ideal solutions for each sector: historians, genealogists, artists, students, researchers, librarians and council staff. We knew we could not solve all the problems but felt sure we could solve some. We needed to work out which problems to address and come up with an achievable solution.

We defined the achievable:

To get public collections accessible by getting them online,

To get private collections online too,

To get the stories out of people’s heads,

To include both historical and contemporary material,

To create a ‘virtual’ exhibition space for artists and craftspeople,

To inspire a workforce of volunteers.

The Solution

The solution was to create a community-built digital library of arts, culture and heritage resources: images, video, audio, documents, web-links, encyclopedia-like articles and discussion threads, with related material clustered together. It would contain both contemporary and historical content. It had to look gorgeous but not intimidating, and it had to behave very cleverly and yet look simple and intuitive. We wanted it to be self-managing and monitoring as far as possible, with no layer of library expertise needed. ‘By the people for the people’ was our mantra. Our community would decide what content they wanted to include and would be able to upload material in any common file format and describe it with common language. It had to facilitate the building and strengthening of relationships, not just between items in Kete, but between people as well. We wanted to use Open source development tools and release it as an Open Source project, adhere to open standards and build an online community to support it.

What is a Kete?

Echoing the Maori proverb of the three baskets, or kete, of knowledge, we called our concept Kete. We really like what the kete represents. We like that they are ‘honest’, practical items, woven from found materials, and that anyone can learn to weave one. We like that they are made from flax, which springs forth from Papatuanuku, the earth mother. We like the link between the flax and the weaver – the person who caressed and shaped the flax into a beautiful or useful object. We like that kete are usually given from one person to another, so linking people together, and that they are usually given to mark an occasion, so there are stories that surround a kete. When a kete is used and taken from one occasion to another, the stories are being told and the history preserved. The kete is an appropriate metaphor for our digital library, and the various types of material it contains.

We dreamed of a country covered with local kete, so that users could search locally or extend their search to their neighbours, or even all kete. We wanted to build something which we could give away.

Development

We managed to source a significant grant and this combined with donations of cash and kind from our community enabled us to get started.

We contacted our friends at Katipo Communications in Wellington, a small web development company, who had helped us develop Koha. They were keen to become involved. We have a long-standing and easy relationship with Katipo and I am sure this helped a lot with developing Kete. We could – and did – have long, loud, excited sessions around a white board or over a curry, hammering out ideas and concepts and relationships and ‘what ifs’, and these resulted in a series of dataflow diagrams which described the processes and functions that we required of Kete.

The Kete team was determined to produce a quality product that did what we said it would, on time and within budget. We had to avoid feature creep as we were working to such a tight budget and timeframe. In thinking big first, and dreaming of how Kete could ultimately look and work, we were able to ensure that the Kete core would contain all the necessary scaffolding for future enhancements. A closer look at Kete Horowhenua can be found in appendix 1.

Is Kete successful?

How do you measure the success of a web 2.0 site? Is it based on the numbers of contributors who actually make use of the web 2.0 features: creating content, making changes, posting comments, emailing other contributors or repurposing the content? How much weight do you attach to the numbers of readers who continue to interact with the material in the old fashioned way – simply reading?

Some months Kete Horowhenua attracts over 450,000 hits from 6,000 unique IP addresses; that is 6,000 individual internet sites counted just once each in the month (so no, not lots of Library staff logging in to increase the figures!) The overwhelming majority of the people who visit the site just look – but increasingly some do participate too.

Web 2.0 is NOT a silver bullet

It is not simply a case of ‘build it and they will come’ though. Launching a web 2.0 application is not the end of the matter. For the first 12 months we created digital content ourselves in order to inspire others, but then we made a conscious decision to step back in order to avoid provider capture. There was a feeling that we were dominating this ‘community’ space with librarian-created content which, while generally of a high quality and gorgeous to look at, was actually quite intimidating to the average user who didn’t want to spoil it by editing or adding to it. This was exactly the opposite of what we were trying to achieve.

We shifted into marketing mode instead. With such a diverse community to try and engage we had to use targeted marketing techniques, devising campaigns for specific sectors within the community. We thought long and hard about the “what’s in it for me” question in relation to different groups: clubs, returned servicemen, genealogists, community event organisers, artists and craftspeople, businesses, historians, the performing arts and schools. We now do a lot of visits in the community delivering audio visual presentations that showcase a very narrow range of Kete content which illustrates how well Kete could help a specific audience achieve its particular goals.

Managing and building an online community takes resources. You need to invest time in the initial stages until it reaches a turning point where it becomes self sustaining. You will need to acknowledge participants – humans want to be noticed! Comment on the new work giving feedback and encouragement, add value to submitted content by making it better or prettier, repurpose submitted content and repost it, promote new work on the front page or in the ‘real’ world through displays, newspaper stories or events. Here Comes Everybody by Clay Shirky is highly recommended but there are lots of resources available online.

Quality in a community repository is a tricky issue. If you make it too hard to ‘play’ by imposing standards or barriers that restrict people they just won’t participate. But not everything added to the site is worth keeping. Who decides and how do you remove inoffensive but essentially unwanted content added to the site by people whom you have invited to contribute?

When considering how successful a web 2.0 application is, I wonder, too, whether it matters who is using the site? What if a large proportion of your audience is international? Is it still a service you can claim to be providing to your local community?, and what is your local community in a borderless world? Is a measure of success in the web 2.0 world how “Google-able” your content is? If you are not visible then do you even exist in a digital world? Think about the BBC’s website content; brilliant material but you can’t find it if you don’t know it exists. Kete content ranks very high, very quickly, in Google searches thanks to the way pages are created by the underlying code.

Bridging the digital divide

In 2001 the UN General Assembly endorsed the holding of the World Summit on the Information Society (WSIS). The objective of the first phase that took place in Geneva 2003 was to “develop and foster a clear statement of political will and take concrete steps to establish the foundations for an Information Society for all”. Representatives from 175 countries attended the Summit, including nearly 50 Heads of state and 82 Ministers.

The first principal of the Geneva Declaration states:

We ….. declare our common desire and commitment to build a people-centred, inclusive and development-oriented Information Society, where everyone can create, access, utilize and share information and knowledge, enabling individuals, communities and peoples to achieve their full potential in promoting their sustainable development and improving their quality of life, premised on the purposes and principles of the Charter of the United Nations and respecting fully and upholding the Universal Declaration of Human Rights.

The WSIS is a powerful and significant driver for initiatives designed to bridge the digital divide which separates our world into the haves and the have-nots. For digital technologies to become pervasive in our societies, breaking through the digital divide, they need to be relevant to people at a personal and local level. They have to provide real value to people's lives and sense of being. Community and belonging are really, really important.

In the last Australian census 22% of Australia’s population were born overseas, about the same ratio as New Zealand, but 50% identified with an ancestry other than Australian and 41% have at least one parent who has born overseas. Cultural identity is an incredibly important contributor to people’s wellbeing and identifying with a particular culture makes people feel they belong and gives them a sense of security. It also provides access to social networks which provide support and shared values and aspirations and has been linked with positive outcomes in areas such as health and education.

Language is important in the transmission of culture to future generations and is a central component to cultural identity, alongside ceremonies, celebrations, customs, cooking practices, arts, crafts, history and experience. In fact language is so important that suppression of the languages of minority groups has historically been used as deliberate policy in order to suppress minority cultures. As recently as the 1960s Playcentre teachers in New Zealand were still advising parents to speak English at home to prepare their children for the ‘world’ and the Hunn Report 1961 described Maori language as "a relic of ancient Maori life"

The concept of community and belonging is especially relevant today, in an age where families, cultures and societies are torn apart for a wide variety of reasons. Kete can be a gathering point for sharing traditional knowledge and history and experiences and memories. Refugees may have nothing of their culture except what they carry in their hearts and heads.

I view history as being a cluster of points of view, a nebulous collection of differing truths. Imagine having a whole range of personal perspectives to sit beside the official authored version of the Russian incursion into Georgian territory, or the experiences ravaging Africa and the Middle East right now.

National Digital Strategy

Following the WSIS New Zealand’s last Labour government produced several significant documents around our digital future as a nation including the NZ Digital Content Strategy (DCS) and the National Digital Strategy (NDS) out of which dropped the Community Partnership Fund (CPF). The CPF was a contestable fund established to support community projects that helped deliver the NDS goals of improving people’s capability and skills in using ICT and developing digital content. Horowhenua Library Trust managed to get 2 grants in subsequent years. The first helped fund the development of Kete Horowhenua while the second funded the release of the generalized software as open source so other communities around the world could set up their own Kete.

Aotearoa People’s Network

The Aotearoa People's Network Kaharoa (APNK) is a key initiative in achieving the goals of the DCS and was also funded from the CPF. It is about providing free access to equipment, training and broadband internet services in public libraries so that all New Zealanders can benefit from creating, accessing and experiencing digital content. The CPF has funded Kete for each of the partnering APNK libraries.

Penny Carnaby, New Zealand’s National Librarian, has a lovely vision of gorgeous local Kete popping up all over New Zealand, and all sitting within a National Digital Library of both formal and informal content. This is exactly what we envisioned over those Indian meals so long ago, each community taking responsibility for huge silos of ‘deeply local’ content but all being seamlessly shared.

Digital NZ

Digital NZ is a collaborative initiative led by the National Library of New Zealand to make New Zealand digital content easy to find, share and use so that we can stay connected to our own stories, creations, knowledge and culture. It includes content from government departments, publicly funded organisations, the private sector, and community groups using repositories like Kete, including all the APNK Kete.

Digital NZ is the ‘magic’ which enables the informal Kete content to sit alongside formal content owned by national organisations.

Open Source

Developing in Open Source was a no brainer for Horowhenua Library Trust. We do not want to be a software development company and we havn’t the resources to solve all the problems ourselves. We were very happy to develop the bit we needed and ensure the scaffolding was in place for future development but we needed to give it away in order to make it better. We are committed to supporting the spread and development of Kete and want a community of developers and users to take it forward.

And forward it has gone.

Kete 1.0 was released as Open Source under a GNU Public License (GPL) at the beginning of June in 2008 and we received lots of inquiries about the Kete Project, both within New Zealand and from overseas. We won a Special Mention for excellent e-content from our region (North America and Oceania) in the e-inclusion section of the World Summit Awards held in Venice late in 2007, and also won the 2007 3M Award for Innovation in New Zealand Libraries.

APNK received funding to offer 34 local authorities their own Kete and about a dozen are up and running so far in New Zealand. Other cities, like Wellington, Upper Hutt and Hamilton have also established Kete for their communities. Kete Hamilton is particularly strong on celebrating the contemporary cultural diversity of the city.

Looking forward

The potential of Kete as a tool for supporting cultural identity is just starting to be realised. The Chinese Digital Community Kete, a joint project by Auckland City Libraries and the NZ Chinese Association, contains historical and contemporary content about New Zealand's Chinese community. It has deeply embedded links to the digital content owned by Auckland Libraries including hundreds of digitized Chinese newspapers and magazines. The site also features an RSS feed of library materials new to the library.

Recent work, including a built in mechanism for creating new translations and the ability to choose left to right or right to left layouts for languages that need that orientation, means that providing Kete in many languages is now possible. This is very important for the Arabic translations under development for King Saud University in Saudi Arabia and the Dimmyatt Public Library in Egypt.

The beauty of developing in open source means that if something is found to be missing in Kete to support a particular language you have the power to make changes to the software to suit your needs. It would be great to have world class support for all kinds of languages in Kete and translation volunteers willing to help us should make contact on the old.kete.net.nz site. There is still a large block of work to be done to get the Zebra search engine supporting non-Latin languages which would result in full translation capabilities. This work will cost $USD20k and co-funders are being sought. Actually the Kete project could do with lots of help, especially tech developers with expertise in Ruby on Rails .

Each of the groups who has a Kete has contributed to the cost of developing enhancements to the core Kete code or towards the configuration interface. This is the beauty of Open Source: development where we all share our ideas for development, we share the cost and we share the benefits. This was also the view taken by CPF who paid for not only Kete Horowhenua and the release of Kete V1.0 but also the Chinese Kete and He Kete Korero : Kete Taranaki Reo which is a digital archive of language, history and traditions of Taranaki Māori. This work is the subject of my co-presenter’s paper and I’ll leave it to Clare Hall to showcase the stunning work going on there.

Conclusion

The translation work that has been carried out already by the Taranaki Reo team, the work being tested now in Arabia and the yet to be funded work to support non–Latin languages are all significant achievements. Kete is on the cusp of becoming a truly useful tool for international and multi cultural use.

Toi tu te kupu,
toi tu te mana,
toi tu te whenua

This proverb was spoken by Tinirau of Wanganui. It is a plead to hold fast to our culture, for without language, without mana (spirit), and without land, the essence of being a Maori would no longer exist, but be a skeleton which would not give justice to the full body of Maoritanga (maoridom).

References

Hunn, J. (1961) Report on Department of Maori Affairs.
Retrieved from http://www.teara.govt.nz/en/urban-maori/1/5

McGinnis, W. & Ransom, J. (2010) Kete and Koha : Integration Built on Open Standards.
OCLC Systems and Services. 26 (2), 114-122.

Millington, R. How to build an Online Community : the Ultimate List of Resources. (2010). Retrieved from http://www.feverbee.com/2010/01/how-to-build-an-online-community-the-ultimate-list-of-resources.html

World Summit on the Information Society (2007) Declaration of Principles.
Retrieved from http://www.itu.int/wsis/docs/geneva/official/dop.html

Appendix 1 : Kete Horowhenua

1.1 The Development Process

Our original plan was to make the most out of our current experience and build a system based on Koha (an open source library management system we developed in 2000) and Greenstone (an open source digital library programme developed by Waikato University). We had a good look at this and, very early on, could see that this was going to be difficult. The Koha/Greenstone approach would work for a more traditional type of collection, where one organisation publishes the collection to a website. But it is more problematic for a community-built collection. Weighing up the time it would take for us to build an add-on for an existing digital library or starting from scratch, we thought that building a system based on a Web Framework might be a better idea. This is where Ruby on Rails comes in.

We had a good look around, and compared to the likes of Catalyst (Perl-based framework) and the PHP-based frameworks, Ruby on Rails (RoR) appeared to have the most effort being put in and the most dynamic community around it. We are attracted to the idea of using a framework, rather than building everything from scratch. It is also new and shiny!

Kete uses Zebra, a full-text indexing engine built by IndexData. This facilitates very fast and very powerful searching over a text based database. It can also handle anything from a few records to many millions. The Koha project has tested the Zebra engine with databases of up to 10 million records. We knew Kete could go live with 10,000 images, and that we were likely to have 20,000 images 6 months post launch, and this was just one item type; there are also documents, audio, video and topics. Zebra is supported natively with Koha Version 3.0 and Koha version 2.4 includes Zebra support via a plugin. This means it should be easier to do a powerful federated search across Koha and Kete – something important to Horowhenua Library Trust, and to Koha libraries around the world.

The development method we used meant that testing occurred throughout development. We used volunteers from our community to ‘test’ the software: school children of many ages but more usefully senior citizens who have only a little experience with web applications. Three instances of the Kete software were operational on the server at any one time: a development site, a test site and a stable, live site. This meant that development was very fast, and that Library staff and volunteers were intrinsically involved in the development process, testing each new bit while the next bit was being written. We concentrated on core functions and proof of concept before writing the next bit, i.e. making a process work for one item type then moving on to the next process.

1.2 What you can do

Kete utilises a range of Web 2.0 technologies. In a nutshell, Web 2.0 is about active participation with a website rather than being a passive observer. In Kete, users can:

Search by keywords, or just browse

View linked items and topics

Start or join a discussion around a record

Register for RSS feeds to keep abreast of changes to records you are interested in.

Contact other people with similar interests by email.

Edit records by fixing errors, adding more information

‘Tag’ records by typing in keywords to help other people find an item or topic.

Upload new items

Add relevant websites.

Write new topics or stories

Link items together into meaningful subject clusters

1.3 Organisation and vocabulary

Kete Horowhenua is a digital library comprising of range of different records, including ITEMs and TOPICs. ITEMs are files uploaded and ‘catalogued’ in common language by Kete users. They may be in any common file format:

images

documents

audio

video

web-links

discussion comments

Items may be standalone, or LINKED together into meaningful clusters with a TOPIC. A Topic is like an encyclopedia entry about any subject, person, place, event or thing that someone wants to write about. Items may be linked to any number of Topics.

Topics are built by selecting a template.

Each Topic Template has a number of suggested

fields which help guide the writer of the

topic, but very few are mandatory. The Topic

Templates are hierarchical in nature meaning

that each sub topic inherits the fields of its

predecessor.

Contributors can select from a range of creative

commons licenses, or accept the site default, and

contributions are accepted on condition that

they may be edited or added to by other Kete

users, except for locked baskets. These are special

‘Kete within a Kete”. These are administered by

their Owners, have their own ‘homepage’ and

Items and Topics are protected from being edited

by unauthorised users, although general Kete users

can view the contents, start or contribute to a

discussion thread, or send an email to the Owner.

Locked baskets are set up on application, and in

special instances only. We have three Locked

baskets in Kete Horowhenua: Te Kokiri,

Adopt an Anzac Project and Trevor Heath Photography.

Te Kokiri is open to all of its students and tutors, who are able to build online portfolios of their work. The Adopt an Anzac Project has verified all of its content against primary sources, and requires any alterations and additions to be similarly verified before uploading changes to the site. Trevor Heath is a professional photographer who has an online portfolio of images from throughout the Horowhenua District. These are grouped together in his Kete by event and are uploaded as low resolution files; he is happy for anyone to look at them online and use them for school projects etc, but they are no use for reproduction purposes.

1.4 Kete Text editor

The description field for Topics and Items can be extensively formatted with the Kete text editor. This allows for the inclusion of tables, hot links and images. This Topic shows how tables have been used to clarify relationships between individuals and generations. Where related Topics exist in Kete they are shown as hotlinks.

Fig 2: Topic detail showing the use of tables, hotlinks and topic–templates.

1.5 Homepage

The Kete Horowhenua homepage provides a number of access points to the digital library to enhance discoverability. We discovered early on that the simple keyword search box on a near empty screen paralysed searchers who did not know what was in Kete or what to search for. A comprehensive help manual is also available from the homepage.

Browse

By clicking on the browse button, you can see the entire contents of the database with each item type displaying on a separate tab. You can vary how many records display per page, which makes it a pretty quick task to scan a few hundred items in a few minutes. This is an easy way to track new additions to the database; the lists are sorted with the newest or edited ones displaying first. You can also access this list from the Contents on Type list on the homepage sidebar.

Featured Topics

These are a changing selection of topics displaying on the homepage. These may be chosen for a number of reasons: they might be new, or interesting, or demonstrate an excellent use of the Kete Text editor.

Slideshow

The homepage also has a continuous slideshow of random images from the database. They change every 20 seconds but a Kete user can click on the image to go to the detailed screen. 

Latest 5 Topics

We have a dynamic list of the newest five topics to be added to the Kete database displaying on the homepage. This promotes the range of material being added.

Keyword Searching

Kete is searched by keywords. The search engine accommodates truncations and simple Boolean operators to define relationships between search terms and phrases. Kete uses fuzzy searching.

Search Results
Search results are sorted by item or file type, with each item type on its own 'tab' displaying the item count. Tabs in Kete work like tabbed divider pages in a ringbinder, separating items into different categories. You can view specific Item types by clicking on the appropriate tab.

McGinnis, (2010).

Millington, (2010).

World Summit on the Information Society, (2007)

Hunn, (1961)

Fig 1: Topic Templates to select from.

<!--Section Ends-->