San Jose, years before it became a town was a wilderness, a veritable hunting ground for wild animals by the inhabitants of its neighboring towns. The earliest inhabitants were known as Negritos (Baluga) headed by Kapitan Danding, a Negrito convert residing in Pinagcuartelan. These early inhabitants depended mostly on hunting and fishing for their livelihood. Some hunters from the neighboring towns found San Jose as a good place for settlers due to its wide and uncleared agricultural land. The first group of settlers made some clearings (kaingin) in the outskirts of the present town.
At the early part of the American regime, peace and order were enjoyed by the inhabitants of the town for a short period of time. In 1902, there was an uprising by an organization named "Sta. Iglesia" headed by a notorious bandit, "Apo Ipe" (Felipe Salvador) who hid in the environs of San Jose. The aim of this organization was the nightmare of the town until Sgt. Gregorio Cadhit of the constabulary detachment of San Jose killed the notorious bandit leader.