Skip to content

D
Distributes-Search
Commit e39b8a01 authored by mohammad.salama's avatar mohammad.salama
Browse files
Options

Fourth Commit - Cluster only - Testing Reading Files

parent a9a99f8f
Hide whitespace changes
Inline Side-by-side
Showing with 142 additions and 59 deletions
.idea/workspace.xml
View file @ e39b8a01
......@@ -10,19 +10,13 @@
</component>
<component name="ChangeListManager">
<list default="true" id="b8bcd35c-99f3-44c8-a866-59f81c3c8bd0" name="Changes" comment="">
<change afterPath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/FileWordPair.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/.idea/workspace.xml" beforeDir="false" afterPath="$PROJECT_DIR$/.idea/workspace.xml" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/Application.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/Application.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/Coordinator.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/Coordinator.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/LeaderElection.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/LeaderElection.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/OnElectionAction.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/OnElectionAction.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/ServiceRegistry.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/ServiceRegistry.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/TransientWorker.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/TransientWorker.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/TransientWorkerApplication.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/AutoHealerAndClusterSearch/TransientWorkerApplication.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/GRPC/grpcClient.java" beforeDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/GRPC/grpcServer.java" beforeDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/GRPC/service.proto" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/GRPC/service.proto" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/Response.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/Response.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/FileWordPair.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/FileWordPair.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/Request.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/SearchQueryRequest.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/Response.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/ObjectExchangeInCluster/SearchQueryResponse.java" afterDir="false" />
<change beforePath="$PROJECT_DIR$/src/main/java/generalTesting/test.java" beforeDir="false" afterPath="$PROJECT_DIR$/src/main/java/generalTesting/test.java" afterDir="false" />
</list>
<option name="SHOW_DIALOG" value="false" />
<option name="HIGHLIGHT_CONFLICTS" value="true" />
......
SearchFiles/f (1).txt
View file @ e39b8a01
Curiosity is a powerful driving force that fuels the flame of learning. It is a natural human inclination to wonder, question, and seek knowledge about the world around us. Curiosity sparks our desire to explore, discover, and understand the mysteries that lie beyond our current understanding. It is through curiosity that we embark on intellectual adventures, pushing the boundaries of what we know and delving into the realm of the unknown. Curiosity drives us to ask "why?" and "how?" and compels us to seek answers, to engage in lifelong learning. It encourages us to embrace new ideas, perspectives, and experiences, enabling personal growth and development. With curiosity as our guiding compass, we open ourselves up to endless possibilities, unlocking the doors to innovation, creativity, and deeper insights. Embrace your curiosity, for it is the gateway to a world of knowledge waiting to be explored.
\ No newline at end of file
SearchFiles/f (2).txt
View file @ e39b8a01
Kindness is a universal language that transcends cultural, social, and linguistic barriers. It has the remarkable ability to touch the lives of others in profound ways. A simple act of kindness can brighten someone's day, restore hope, and inspire positive change. Whether it's a warm smile, a helping hand, or a compassionate ear, the power of kindness lies in its ability to create connections and foster a sense of belonging. In a world that often feels divided, kindness serves as a powerful catalyst for building bridges, spreading love, and making the world a better place, one small act at a time.
\ No newline at end of file
SearchFiles/f (3).txt
View file @ e39b8a01
Nature is a breathtaking masterpiece that surrounds us in its awe-inspiring beauty. From majestic mountains to cascading waterfalls, from vibrant flowers to serene forests, nature captivates our senses and nourishes our souls. It offers solace, tranquility, and a reminder of the interconnectedness of all living beings. In the midst of our busy lives, taking a moment to appreciate the wonders of nature can bring about a sense of wonder, humility, and gratitude. Nature is a constant reminder of the intricate balance and resilience of our planet, urging us to protect and preserve its precious gifts for generations to come.
\ No newline at end of file
SearchFiles/f (4).txt
View file @ e39b8a01
Music is a universal language that resonates deep within our hearts and souls. It has the power to evoke emotions, stir memories, and transport us to different times and places. Whether it's the soothing melody of a lullaby, the uplifting rhythm of a dance tune, or the soul-stirring harmonies of a symphony, music has the ability to uplift, heal, and inspire. It transcends barriers and brings people together, fostering a sense of unity and shared experiences. Whether you're a listener, a performer, or a creator, the joy of music is boundless, offering a soundtrack to our lives and a source of endless wonder and delight.
\ No newline at end of file
SearchFiles/f (5).txt
View file @ e39b8a01
In a fast-paced world filled with distractions, mindfulness offers an oasis of calm and presence. It is the practice of paying attention to the present moment, without judgment, and with a gentle curiosity. Mindfulness invites us to fully experience each moment, to savor the simple pleasures, and to cultivate a deeper sense of self-awareness. Through mindfulness, we can find refuge from the constant chatter of the mind and connect with our inner wisdom and intuition. It allows us to embrace life's ups and downs with greater resilience and to cultivate a profound sense of gratitude for the beauty and richness of each passing moment.
\ No newline at end of file
SearchFiles/f (6).txt
View file @ e39b8a01
Books are portals that transport us to different worlds, ignite our imagination, and expand our horizons. They hold within their pages the collective wisdom, stories, and experiences of humanity. Whether it's a gripping novel that keeps us on the edge of our seats, a thought-provoking non-fiction book that challenges our perspectives, or a poetic masterpiece that stirs our souls, books have the power to educate, entertain, and inspire. They offer solace during difficult times, companionship in moments of solitude, and a gateway to knowledge that knows no boundaries. In a world of endless distractions, the magic of books lies in their ability to evoke emotions, spark curiosity, and leave an indelible mark on our lives.
\ No newline at end of file
SearchFiles/f (7).txt
View file @ e39b8a01
Gratitude is a transformative practice that shifts our focus from what is lacking to what is abundant in our lives. It is a powerful reminder to appreciate the simple joys, blessings, and acts of kindness that surround us each day. Cultivating gratitude allows us to find beauty in the ordinary, to cherish the relationships that enrich our lives, and to embrace adversity as an opportunity for growth. It fosters a positive mindset, enhances our well-being, and strengthens our connections with others. Gratitude is a gentle reminder to pause, reflect, and give thanks for the precious gift of life and all that it encompasses.
\ No newline at end of file
SearchFiles/f (8).txt
View file @ e39b8a01
Resilience is the unwavering spirit that emerges in the face of adversity. It is the ability to bounce back, adapt, and grow stronger from life's challenges and setbacks. Resilience resides within each of us, waiting to be awakened and nurtured. It is a testament to the human spirit's remarkable capacity for growth and transformation. In difficult times, resilience guides us through the darkest of tunnels and reminds us that there is light at the end. It empowers us to embrace change, overcome obstacles, and find new paths forward. Resilience is a reminder that even in the face of adversity, we have the strength to rise, rebuild, and thrive.
\ No newline at end of file
src/main/java/AutoHealerAndClusterSearch/Coordinator.java
View file @ e39b8a01
package AutoHealerAndClusterSearch;
import ObjectExchangeInCluster.FileWordPair;
import ObjectExchangeInCluster.Request;
import ObjectExchangeInCluster.Response;
import ObjectExchangeInCluster.SearchQueryRequest;
import ObjectExchangeInCluster.SearchQueryResponse;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;
......@@ -62,9 +62,9 @@ public class Coordinator
query = scanner.nextLine();
Thread clientThread = new Thread(() ->
{
Response responseMap = handleClient(query);
SearchQueryResponse searchQueryResponseMap = handleClient(query);
printResponsr(responseMap);
printResponsr(searchQueryResponseMap);
//sendResponsesToClient(clientSocket, responseMap);
});
......@@ -74,34 +74,34 @@ public class Coordinator
}
private void printResponsr(Response responseMap)
private void printResponsr(SearchQueryResponse searchQueryResponseMap)
{
}
private Response sendRequestToNode(Request request, String ipAddress)
private SearchQueryResponse sendRequestToNode(SearchQueryRequest searchQueryRequest, String ipAddress)
{
Response response = null;
SearchQueryResponse searchQueryResponse = null;
String ip = ipAddress.split(":")[0];
int port = Integer.parseInt(ipAddress.split(":")[1]);
try (Socket socket = new Socket(ip, port))
{
ObjectOutputStream objectOutputStream = new ObjectOutputStream(socket.getOutputStream());
objectOutputStream.writeObject(request);
objectOutputStream.writeObject(searchQueryRequest);
// Receive the response
ObjectInputStream objectInputStream = new ObjectInputStream(socket.getInputStream());
response = (Response) objectInputStream.readObject();
searchQueryResponse = (SearchQueryResponse) objectInputStream.readObject();
} catch (IOException | ClassNotFoundException e) {
e.printStackTrace();
}
return response;
return searchQueryResponse;
}
private void sendResponsesToClient(Socket clientSocket, Response response)
private void sendResponsesToClient(Socket clientSocket, SearchQueryResponse searchQueryResponse)
{
try {
ObjectOutputStream objectOutputStream = new ObjectOutputStream(clientSocket.getOutputStream());
......@@ -116,12 +116,12 @@ public class Coordinator
private Response handleClient(String query)
private SearchQueryResponse handleClient(String query)
{
try
{
List<Response> responses = spreadQuery(query);
processResultsAndPrepareAnswer(responses);
List<SearchQueryResponse> respons = spreadQuery(query);
processResultsAndPrepareAnswer(respons);
}
catch (IOException | InterruptedException | KeeperException e)
{
......@@ -144,11 +144,11 @@ public class Coordinator
* and values are list of pair<fileName,Number Of Appearance of the word in it>
* we now must calculate the importance of each file for every word we got
* */
private void processResultsAndPrepareAnswer(List<Response> responses)
private void processResultsAndPrepareAnswer(List<SearchQueryResponse> respons)
{
for (Response response : responses)
for (SearchQueryResponse searchQueryResponse : respons)
{
TreeMap<String , List<FileWordPair>> treeMap = response.getWordFrequencies();
TreeMap<String , List<FileWordPair>> treeMap = searchQueryResponse.getWordFrequencies();
for(Map.Entry<String, List<FileWordPair>> mapEntry : treeMap.entrySet())
{
System.out.println("The word " + mapEntry.getKey() + " is :");
......@@ -188,7 +188,7 @@ public class Coordinator
* response is tree-map here the keys are words
* and values are list of pair<fileName,Number Of Appearance of the word in it>
* */
public List<Response> spreadQuery(String query) throws InterruptedException, KeeperException, IOException, ExecutionException
public List<SearchQueryResponse> spreadQuery(String query) throws InterruptedException, KeeperException, IOException, ExecutionException
{
List<String> physicalZnodes = zooKeeper.getChildren(PHYSICAL_ZNODES_PATH, false);
......@@ -200,7 +200,7 @@ public class Coordinator
int filesOffset=0;
ExecutorService executorService = Executors.newFixedThreadPool(physicalZnodes.size());
List<Callable<Response>> tasks = new ArrayList<>();
List<Callable<SearchQueryResponse>> tasks = new ArrayList<>();
///distributing Files for NODES!!!
while (remaining > 0)
......@@ -219,24 +219,24 @@ public class Coordinator
PHYSICAL_ZNODES_PATH + "/" + physicalZnodes.get(index),
false, stat)
);
Request request = new Request(query , filesNumberforNode , filesOffset);
tasks.add(() -> sendRequestToNode(request, ipAddress));
SearchQueryRequest searchQueryRequest = new SearchQueryRequest(query , filesNumberforNode , filesOffset);
tasks.add(() -> sendRequestToNode(searchQueryRequest, ipAddress));
index = (1 + index) % physicalZnodes.size();
filesOffset = (filesOffset + filesNumberforNode) % totalFilesNumber;
remaining -= filesNumberforNode;
}
List<Future<Response>> futures = executorService.invokeAll(tasks);
List<Future<SearchQueryResponse>> futures = executorService.invokeAll(tasks);
List<Response> responses = new ArrayList<>();
List<SearchQueryResponse> respons = new ArrayList<>();
for (int i = 0; i < futures.size(); i++)
{
Response response = futures.get(i).get();
responses.add(response);
SearchQueryResponse searchQueryResponse = futures.get(i).get();
respons.add(searchQueryResponse);
}
executorService.shutdown();
return responses;
return respons;
}
......
src/main/java/AutoHealerAndClusterSearch/TransientWorker.java
View file @ e39b8a01
package AutoHealerAndClusterSearch;
import ObjectExchangeInCluster.Request;
import ObjectExchangeInCluster.Response;
import ObjectExchangeInCluster.SearchQueryRequest;
import ObjectExchangeInCluster.SearchQueryResponse;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.ZooDefs;
......@@ -65,11 +65,11 @@ public class TransientWorker
Thread clientThread = new Thread(() ->
{
Response responseMap = null;
SearchQueryResponse searchQueryResponseMap = null;
try
{
responseMap = handleCoordinator(clientSocket);
sendResponsesToClient(clientSocket, responseMap);
searchQueryResponseMap = handleCoordinator(clientSocket);
sendResponsesToClient(clientSocket, searchQueryResponseMap);
} catch (IOException | ClassNotFoundException e) {
throw new RuntimeException(e);
}
......@@ -80,23 +80,23 @@ public class TransientWorker
}
}
private void sendResponsesToClient(Socket clientSocket, Response responseMap)
private void sendResponsesToClient(Socket clientSocket, SearchQueryResponse searchQueryResponseMap)
{
}
private Response handleCoordinator(Socket clientSocket) throws IOException, ClassNotFoundException
private SearchQueryResponse handleCoordinator(Socket clientSocket) throws IOException, ClassNotFoundException
{
Request request = extractRequest(clientSocket);
double score = getScore(request , getMyFiles(request.getNumberOfFilesToScan() , request.getFilesOffset()));
SearchQueryRequest searchQueryRequest = extractRequest(clientSocket);
double score = getScore(searchQueryRequest, getMyFiles(searchQueryRequest.getNumberOfFilesToScan() , searchQueryRequest.getFilesOffset()));
return null;
}
private Request extractRequest(Socket clientSocket) throws IOException, ClassNotFoundException
private SearchQueryRequest extractRequest(Socket clientSocket) throws IOException, ClassNotFoundException
{
ObjectInputStream objectInputStream = new ObjectInputStream(clientSocket.getInputStream());
Request request = (Request) objectInputStream.readObject();
return request;
SearchQueryRequest searchQueryRequest = (SearchQueryRequest) objectInputStream.readObject();
return searchQueryRequest;
}
private List<String> getMyFiles(int numberOfFilesToScan, int filesOffset)
......@@ -120,7 +120,7 @@ public class TransientWorker
return fileNames.subList(startIndex, endIndex);
}
private double getScore(Request query, List<String> myFiles)
private double getScore(SearchQueryRequest query, List<String> myFiles)
{
String[] wordsToCount = query.getQueryWords();
Map<String , BigInteger> numberOfWordsInFile = new HashMap<>();
......
src/main/java/ObjectExchangeInCluster/FileWordPair.java
View file @ e39b8a01
......@@ -5,6 +5,11 @@ public class FileWordPair
public String fileName;
public long freq;
public FileWordPair (String fileName , long freq)
{
this.fileName = fileName;
this.freq = freq;
}
@Override
public String toString()
{
......
src/main/java/ObjectExchangeInCluster/Request.java → src/main/java/ObjectExchangeInCluster/SearchQueryRequest.java
View file @ e39b8a01
......@@ -2,12 +2,12 @@ package ObjectExchangeInCluster;
import java.io.Serializable;
public class Request implements Serializable
public class SearchQueryRequest implements Serializable
{
private String query;
private int numberOfFilesToScan;
private int filesOffset;
public Request(String query , int numberOfFilesToScan , int filesOffset)
public SearchQueryRequest(String query , int numberOfFilesToScan , int filesOffset)
{
this.numberOfFilesToScan = numberOfFilesToScan;
this.filesOffset = filesOffset;
......
src/main/java/ObjectExchangeInCluster/Response.java → src/main/java/ObjectExchangeInCluster/SearchQueryResponse.java
View file @ e39b8a01
......@@ -5,22 +5,23 @@ import java.util.ArrayList;
import java.util.List;
import java.util.TreeMap;
public class Response implements Serializable {
public class SearchQueryResponse implements Serializable {
private TreeMap<String, List<FileWordPair>> wordFrequencies;
public Response() {
public SearchQueryResponse() {
wordFrequencies = new TreeMap<>();
}
public void addFileWordPair(String fileName, String word, long freq) {
/* public void addFileWordPair(String fileName, String word, long freq)
{
FileWordPair fileWordPair = new FileWordPair();
fileWordPair.fileName = fileName;
fileWordPair.freq = freq;
List<FileWordPair> temp = wordFrequencies.getOrDefault(word, new ArrayList<>());
temp.add(fileWordPair);
wordFrequencies.put(word, temp);
}
}*/
public TreeMap<String, List<FileWordPair>> getWordFrequencies()
{
......
src/main/java/generalTesting/test.java
View file @ e39b8a01
package generalTesting;
import ObjectExchangeInCluster.FileWordPair;
import java.io.*;
import java.net.Socket;
import java.nio.file.DirectoryStream;
......@@ -14,7 +16,6 @@ public class test
public static void listFilesForFolder(String filesLocation)
{
List<String> fileNames = new ArrayList<>();
try (DirectoryStream<Path> stream =
Files.newDirectoryStream(Paths.get(filesLocation)))
{
......@@ -30,10 +31,84 @@ public class test
for (String file : fileNames) System.out.println(file);
}
public static void countWordsInFiles()
{
String path = System.getProperty("user.dir") + "/SearchFiles/";
List<String> fs = new ArrayList<>();
fs.add(path+"f (1).txt");
fs.add(path+"f (3).txt");
fs.add(path+"f (5).txt");
fs.add(path+"f (6).txt");
String query ;
Scanner scanner = new Scanner(System.in);
query= scanner.nextLine();
String[] ws = query.split(" ");
List<String> words = new ArrayList<>();
for (String w : ws)
{
words.add(w);
}
Map<String, List<FileWordPair>> wordOccurrences = countWordOccurrences(words, fs);
// Print the word occurrences
for (Map.Entry<String, List<FileWordPair>> entry : wordOccurrences.entrySet()) {
String word = entry.getKey();
List<FileWordPair> occurrences = entry.getValue();
System.out.println("Word: " + word);
for (FileWordPair fileOccurrence : occurrences) {
System.out.println("File: " + fileOccurrence.fileName + ", Count: " + fileOccurrence.freq);
}
}
}
public static Map<String, List<FileWordPair>> countWordOccurrences(List<String> words, List<String> filePaths)
{
Map<String, List<FileWordPair>> wordOccurrences = new HashMap<>();
for (String filePath : filePaths)
{
try (BufferedReader reader = new BufferedReader(new FileReader(filePath)))
{
StringBuilder textBuilder = new StringBuilder();
String line;
while ((line = reader.readLine()) != null) {
textBuilder.append(line).append("\n");
}
String text = textBuilder.toString();
// Count occurrences of words in the text
for (String word : words)
{
int count = countWordOccurrencesInText(word, text);
List<FileWordPair> FileWordPair = wordOccurrences.getOrDefault(word, new ArrayList<>());
FileWordPair.add(new FileWordPair(filePath, count));
wordOccurrences.put(word, FileWordPair);
}
} catch (IOException e) {
e.printStackTrace();
}
}
return wordOccurrences;
}
public static int countWordOccurrencesInText(String word, String text)
{
int count = 0;
int index = text.toLowerCase().indexOf(word.toLowerCase());
while (index != -1)
{
count++;
index = text.toLowerCase().indexOf(word.toLowerCase(), index + 1);
}
return count;
}
public static void main(String[] args)
{
///listFilesForFolder(System.getProperty("user.dir") + "/SearchFiles");
countWordsInFiles();
}
}
target/classes/META-INF/MANIFEST.MF
View file @ e39b8a01
Manifest-Version: 1.0
Main-Class: AutoHealerAndLoadBalancing.Application
Main-Class: AutoHealerAndClusterSearch.Application
target/classes/logback.xml
View file @ e39b8a01
......@@ -17,7 +17,7 @@
</encoder>
</appender>
<!-- Log Level: ERROR, WARN, INFO, DEBUG, TRACE-->
<logger name="AutoHealerAndLoadBalancing" level="DEBUG" additivity="false">
<logger name="AutoHealerAndClusterSearch" level="DEBUG" additivity="false">
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</logger>
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

HIAST - GitLab